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ABSTRACT 


This  research  examines  a  process  to  estimate  the  value  of  graduate  education. 
Moreover,  it  demonstrates  an  approach  to  measuring  the  use  of  graduate  education  within 
organizations.  Marine  Corps  officers  who  graduated  from  the  Naval  Postgraduate 
School’s  Information  System  Technology  curriculum  are  studied.  The  study  used  a  web- 
based  survey  for  data  collection  and  a  Knowledge  Value  Added  method  to  objectively 
estimate  the  value  of  education  topics  across  different  Marine  Corps  processes.  Results 
indicate  that  the  Infonnation  System  Technology  curriculum  is  designed  and 
implemented  to  successfully  meet  sponsor  requirements.  It  reveals  that  the  education  is 
highly  valued  and  frequently  used  in  post  graduation  billets.  The  most  valued  aspect  of 
the  education  is  theoretical  knowledge.  However,  the  research  showed  how  practical 
information  technology  skills  and  social  relationships  that  developed  during  the  resident 
education  were  also  highly  valued  and  frequently  used.  The  results  go  on  to  show  that 
personal  interest  in  education  topics  often  corresponded  to  greater  perceived  value. 
Lastly,  a  proof  of  concept  demonstrates  a  method  to  measure  and  compare  the  use  on 
graduate  education  in  subsequent  organizational  processes.  The  Knowledge  Value  Added 
method  provides  the  ability  to  compare  education  use  between  different  topics,  across 
different  jobs,  and  between  different  people. 
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I.  INTRODUCTION 


This  research  examines  a  process  to  estimate  the  value  of  graduate  education  as  it 
contributes  to  core  organizational  processes.  The  Marine  Corps’  process  for  providing  for 
and  employing  graduate  level  education  in  the  information  technology  field  is  examined 
as  a  proof  of  concept.  Application  of  this  model  provides  insight  into  the  relationships 
between  graduate  level  curriculum,  adult  learning  processes,  and  organizational  value. 
An  increased  understanding  of  these  relationships  contributes  to  the  effective 
management  of  educational  investments  as  well  as  to  the  design  and  administration  of 
graduate  level  education  programs. 

A.  LITERATURE  REVIEW 

The  results  of  a  review  of  the  relevant  literature  concerning  the  value  of  graduate 
level  education  can  be  grouped  into  four  categories:  gross  economic  value,  corporate 
return  on  education,  training  management,  and  performance  studies.  These  collections  of 
research  each  seek  appropriate  metrics  and  methods  to  determine  the  value  of  education 
from  diverse  points  of  view. 

1.  Economics  of  Education  and  Human  Capital 

The  economic  theory  of  human  capital  is  based  upon  the  idea  that  human  beings 
are  assets  that  have  some  tangible  value.  It  is  the  human  version  of  physical  capital.  Just 
like  physical  capital,  investments  in  human  capital  can  be  evaluated  based  upon  an 
anticipated  flow  of  future  income  or  on  expected  future  value,  in  non-profit  organizations 
like  the  military. 

Organizations  and  individuals  expend  significant  resources  to  acquire  education. 
The  time,  energy,  and  money  committed  to  graduate  education  presuppose  a  linkage  to 
increased  value  of  the  educated  person.  The  deliberate  commitment  of  resources  to 
achieve  greater  skill  and  knowledge  is  one  of  the  fundamental  premises  of  human  capital 
theory  (Schultz  1961;  Becker  1975).  This  paper  recognizes  this  fundamental  theory  and 
defines  human  capital  as  the  amalgamation  of  a  person’s  skills,  knowledge,  and  talent. 

Educational  investment  is  but  one  type  of  human  capital  enhancement.  The  other 
principal  ways  to  invest  in  human  capital  are  through  health  and  migration  (Kiker  1971). 
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“Educational  investment”  is  used  loosely  in  this  section  to  include  the  broader  concept  of 
learning.  Learning  is  any  process  that  changes  a  participant’s  attitudes,  improves 
knowledge,  and/or  increases  skills  (Kirkpatrick  1998,  20).  These  knowledge,  skills,  and 
abilities  are  often  abbreviated  KSAs. 

The  examination  of  educational  impact  on  human  capital  is  not  new.  This  field  of 
study,  as  we  know  it  today,  was  bom  more  than  thirty  years  ago.  Table  1  presents  the 
prominent  studies  on  the  contribution  of  education  to  human  capital. 


Theory 

Explanation  and  Significance 

Human  capital  investments 
result  in  increased  future 
value  (Becker  1975). 

Presents  one  of  the  foundational  theories  of  human  capital 
investment.  Examines  the  relations  between  earnings,  costs,  and 
rates  of  returns  for  learning  investments. 

Differentiation  between 
public  &  private  labor 
markets  (Hinchliffe  1987). 

Presents  how  internal  labor  markets  in  public  sectors  tend  to  focus 
on  seniority,  education,  and  cultural  practices  as  opposed  to 
productivity,  which  is  more  dominant  in  private  markets. 

Self-selection  bias  (Rosen 
1987). 

Presents  a  theory  on  the  influence  of  personal  choice  in  academic 
participation.  Examines  the  difficulty  of  comparing  the  value  of 
education  based  on  the  variably  of  students  who  participate.  The 
pool  of  graduate  students  is  not  typical  of  the  general,  non¬ 
graduate,  population. 

Human  capital  investment 
processes  and  investment 
incentives  (Shultz  1971). 

Presents  a  theory  of  human  capital  investment  that  addresses  the 
fundamental  comparison  between  investing  in  men  and  investing 
human  capital. 

Exploration  of 
shortcomings  in  earning 
based  approach  to 
measuring  educational 
value  (Solmon  1987). 

Presents  a  range  of  variables  that  diminishes  the  validity  of 
earning  based  measurements  of  educational  impact. 

Screening  theory 
(Woodhall  1987a,  1987b). 

Presents  how  the  organizational  value  of  educated  workers  is  not 
solely  based  on  academic  achievement.  Shows  that  academic 
achievement  is  often  used  as  a  filter  to  screen  for  human  potential, 
independent  of  actual  educational  performance. 

Table  1.  Studies  of  Educational  Investments  in  Human  Capital 


The  authors  highlighted  in  Table  1  have  made  significant  contributions  to  the 
study  of  human  capital.1  They  have  advanced  the  general  awareness  and  depth  of  study  in 
this  field,  and  their  works  have  advanced  the  foundational  theories  of  human  capital. 


1  Schultz  (1979)  and  Becker  (1992)  received  the  Nobel  Prize  in  Economics  for  their  work  in  this  field. 
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These  theories  provide  useful  apparatus  to  analyze  the  general  relationships  between 
learning  (as  well  as  other  factors)  and  human  capital.  However,  these  theories  alone  do 
not  provide  for  the  detailed  analysis  required  to  make  specific  decisions  concerning 
organizational  investments  in  graduate  education. 

2.  Corporate  Return  on  Investment 

One  attempt  to  provide  decision  makers  with  useful  tools  to  administer  training 
and  education  programs  has  been  advanced  by  the  American  Society  for  Training  and 
Development  (ASTD).  Founded  in  1944,  ASTD  has  served  as  one  of  the  world’s  leading 
professional  associations  dedicated  to  the  study  of  workplace  training  and  education.  In 
1997,  ASTD  began  providing  organizations  with  standard  definitions  and  metrics  for 
measuring  and  valuing  their  investments  in  personnel  erudition.  In  return  for  providing 
ASTD  their  training  and  education  data,  organizations  receive  benchmarking  information 
that  compares  their  investments  across  several  reference  groups  (ASTD  2002). 

The  ASTD  benchmarking  database  included  training  and  education  data  from 
over  2500  firms.  The  data  include  such  fields  as  training  dollars  per  employee,  type  of 
training,  and  training  delivery  method.  Over  sixty  percent  of  the  firms  are  headquartered 
in  the  United  States.  Of  the  American  firms,  575  are  publicly  traded.  For  these  firms, 
training  and  education  data  are  compared  to  the  firms’  subsequent  year  Total  Stockholder 
Return  (TSR).2 3  In  other  words,  ASTD  examined  how  training  investments  in  one  year 
influenced  TSR  in  the  following  year  (Bassi  2000). 

The  analysis  of  data  from  1996,  1997,  and  1998  indicate  that  companies  in  the  top 
quartile  of  training  investment  have  higher  median  TSRs  the  following  year;  see  Figure  1. 

This  benchmarking  effort  suggests  a  possible  connection  between  investments  in 
employee  learning  and  a  firms’  net  value.  Presumably,  the  application  of  greater 
knowledge  spawns  increased  innovation,  efficiency,  and/or  productivity  resulting  in  the 
firms’  increased  value.  A  correlation  between  employee  learning  and  capital  gains  enable 
private  organizations  to  justify  their  expenditures  on  employee  education.  Unfortunately, 
this  benchmarking  effort  has  not  been  applied  to  non-profit  organizations  because  it  is 
tied  to  stockholder  return. 

2  TSR  represents  the  change  in  stock  price  plus  issued  dividends  for  a  given  year. 
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Training  expenditure/employee  quartiles  (4=high) 


Figure  1.  Training  and  Developement  Benchmarking  Data  (From:  Bassi  2000) 

The  results  of  this  effort  support  an  intuitive  notion  that  investing  in  employee 
learning  should  result  in  increased  organization  value,  but  this  study  provides  no  insight 
into  how  learning  investments  affect  sub-corporate  entities.  Because  TSR  represents  an 
aggregation  of  corporate  data,  the  ASTD  study  does  not  provide  a  correlation  between 
learning  investment  and  organizational  output  below  a  corporate  level.  This  ability  to 
measure  educational  impact  at  sub-corporate  levels  is  particularly  required  to  target 
learning  investments  crafted  to  achieve  precise  process  improvements  by  the  educated 
employee. 

Furthermore,  this  study  is  not  able  to  show  how  changes  in  TSR  are 
unambiguously  related  to  investments  in  employee  training  and  education.  The 
computation  of  benchmarking  data  assumes  that  employee  learning  is  the  major  cause  of 
TSR  change.  This  approach  fails  to  recognize  other  factors  that  may  also  affect  TSR, 
such  as  changes  in  personnel  acquisition  or  downsizing,  business  practices  and  policy, 
and/or  market  trends. 

The  ASTD  benchmarking  project  advances  the  practical  aspects  to  calculating 

corporate  return  on  learning  investment.  But  by  design,  it  fails  to  relate  learning 

investments  to  sub-corporate  level  returns  and  does  not  provide  the  metrics  to  gauge 

return  on  education  in  non-profit  organizations.  Yet,  the  ability  to  manage  individual 

investments  and  returns  of  employee  education  and  training  remains  a  high-priority 
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capability  required  by  managers.  This  capability  allows  decision  makers  to  optimize  and 
justify  investments  in  employee  education  and  training  against  other  competing 
investment  requirements. 

3.  Managing  Training  as  a  Human  Resource 

Providing  decision  makers  with  tools  and  processes  designed  to  manage  an 
organization’s  human  capital  assets  is  one  of  the  basic  objectives  of  Human  Resource 
Management  (HRM).  Human  Resource  Management  encompasses  many  fields  including 
recruitment  and  placement,  training  and  development,  compensation  and  motivation, 
appraisal  and  career  management,  and  legal  considerations  (Dessler  1991).  Each  of  these 
topics  is  supported  by  large  bodies  of  research  that  deals  with  strategies  geared  towards 
improving  organizational  efficiencies.  The  following  section  reviews  current  trends  and 
studies  in  HRM  training  and  development  as  it  relates  to  learning  investments  and 
potential  returns. 


a.  Defining  Learning 

The  phrase  training  and  development  are  used  in  much  of  the  human 
resource  literature,  while  training  and  education  are  more  commonly  used  in  the  military. 
In  practical  terms,  both  phrases  describe  a  similar  process — learning. 

Training  and  education  are  two  of  the  most  commonly  recognized 
endeavors  to  increase  human  capital.  Often  combined  or  used  interchangeably,  training 
and  education  usually  differ  in  their  processes  and  goals.  Except  where  noted,  this  paper 
uses  the  Marine  Corps’  definitions  of  training  and  education,  as  defined  below  (Marine 
Corps  1999b). 

Training  is  the  building  in  of  information  and  procedures;  using  the 
progressive  repetition  of  tasks,  the  product  of  training  is  skill  development 
and  proficiency.  Training  is  performance  based  and  is  typically  measured 
by  objective  standards. 

Education  ...  is  the  drawing  out  of  students  to  initiate  the  learning  process 
and  bring  their  own  interpretations  and  energies  to  bear — the  product  of 
which  is  a  creative  mind.  Educational  objectives  may  be  measured 
directly,  but  are  often  inferred  from  subjective  testing  or  a  sampling  of 
student  behavior  over  a  period  of  time. 
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The  main  difference  in  these  definitions  emerges  from  the  type  of 
knowledge  they  seek  to  develop.  Military  training  is  generally  geared  towards  increasing 
explicit  procedural  knowledge  and  reinforcing  reliable  application  of  skills.  The  desired 
outcome  of  training  is  consistent  performance  measured  against  established  standards. 
Conversely,  education  seeks  to  instill  an  increased  ability  to  apply  concepts  and  skills  in 
unstructured  and  unfamiliar  situations.  The  results  of  education  tend  to  be  more  implicit 
and  more  difficult  to  define  as  compared  to  training  objectives. 

When  training  and  education  are  forged  with  experience,  a  more  complete 
model  of  learning  is  created.  Figure  2  presents  a  relationship  of  these  three  actions.  This 
simple  model  identifies  and  relates  the  principle  actions  and  outcomes  of  learning. 
Training  tends  to  produce  skills.  The  learner’s  experience  provides  greater  context  for  the 
utilization  of  skills.  Education  advances  the  learner’s  awareness  of  cognitive  concepts. 
While  each  of  these  elements  increase  a  learner’s  human  capital;  collectively,  they 
provide  a  synergy  to  achieve  greater  human  potential. 


Figure  2.  Learning  Model  (After:  Oser  2002) 


b.  Training  Evaluation 

Human  resource  processes  designed  to  measure  the  value  of 
organizational  training  are  relevant  to  this  study  and  are  reviewed  in  this  section. 
Notwithstanding  that  human  resource  management  primarily  focuses  on  training  as 
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opposed  to  education,  the  frameworks  used  to  measure  training  value  may  also  be 
appropriate  to  estimating  education  value. 

Donald  Kirkpatrick  and  Jack  Phillips  are  recognized  in  the  field  of  human 
resource  management  as  two  of  the  principal  contributors  of  techniques  to  evaluate 
organizational  training.  Kirkpatrick  was  the  first  to  articulate  a  model  to  assess  the 
effectiveness  of  training  (Kirkpatrick  1975). 3  In  the  1990s,  Phillips  dominated  much  of 
the  human  resource  development  literature  when  he  advanced  Kirkpatrick’s  four-level 
model.  He  promoted  a  fifth  level  of  evaluation  to  detennine  the  impact  of  training.  He 
called  this  fifth  level  of  evaluation,  training  Return  On  Investment  (ROI). 

There  are  several  strategies  used  to  evaluate  training.  Table  2  presents  four 
of  the  most  prominent  models.  Examining  these  strategies  reveals  similarities  in  the 
evaluation  approaches  and  objectives.  They  generally  provide  for  end-of-training 
evaluations  of  the  training  program  by  the  trainees  in  order  to  evaluate  the  administration 
of  the  training.  They  seek  to  detennine  the  degree  of  training  transfer — what  the  trainee 
actually  learned — as  a  means  of  isolating  the  effects  of  the  training.  Lastly,  to  varying 
extents,  they  try  to  determine  the  impact  of  the  training  within  an  organization  or  process 
in  order  to  make  more  infonned  resource  allocation  decisions.  Phillips’  five-level  model 
is  the  most  advanced  and  emphasizes  the  fifth-level — calculation  of  a  monetary  ROI  from 
training.  The  remainder  of  this  section  focuses  on  Phillips’  model  and  specifically 
examines  his  process  to  calculate  a  return  on  learning  investment. 


3  This  model  was  presented  in  a  series  of  four  articles  in  the  Journal  of  the  American  Society  for  Training 
and  Development ,  November  1959  through  February  1960. 
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Four-Level  Approach 

(Kirkpatrick  1998,  19- 
24) 

Five  Levels  of 
Evaluation  (Kaufman 
and  Keller  1994) 

Context,  Input, 
Reaction,  & 
Outcome  Approach 

(Phillips  1997,  40-41) 

Five-Level  ROI 
Framework  (Phillips 
1997,  42-43) 

1.  Reaction 

2.  Fearning 

3.  Behavior 

4.  Results 

la.  Enabling 

lb.  Reaction 

2.  Acquisition 

3.  Application 

4.  Organizational 
Output 

5.  Social  Outcomes 

1 .  Context  Evaluation 

2.  Input  Evaluation 

3.  Reaction  Evaluation 

4.  Outcome 

Evaluation 

1 .  Reaction  &  Planned 
Action 

2.  Learning 

3.  Job  Application 

4.  Business  Results 

5.  Return  on 

Investment 

Table  2.  Training  Evaluation  Models 


c.  Calculating  Return  on  Training  Investment 
To  advance  his  training  evaluation  model,  Phillips  focused  upon 
articulating  and  advancing  the  fifth-level  ROI  step.  This  step  represents  a  significant 
contribution  to  Kirkpatrick’s  classic  four-level  evaluation  process.  Phillips’  ROI  process 
model  is  presented  in  Figure  3. 


Figure  3.  Training  Return  on  Investment  Process  Model  (From:  Phillips  1998) 
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The  four  ovals  in  Phillips’  ROI  process  model,  shown  in  Figure  3,  denote 
planning  decisions  required  to  prepare  for  ROI  calculations.  These  four  elements  help 
shape  how  and  what  data  are  collected  to  begin  the  ROI  calculation  process.  The  data 
collection  for  this  process  is  non-specific  and  can  consist  of  any  variety  of  methods 
ranging  from  surveys,  to  observations,  to  focus  groups,  as  well  as  others  methods. 

Once  data  is  collected,  Phillips  describes  strategies  to  help  isolate  the 
effects  of  the  training  (Phillips  1998).  These  tend  to  be  subjective  assessments  by 
participants,  managers,  and/or  customers;  and  they  generally  describe  the  effects  of 
training  in  such  terms  of  changes  in  productivity,  service,  and  quality. 

The  third  step  in  the  ROI  model,  shown  in  Figure  3,  involves  converting 
collected  data  to  monetary  value.  This  step  is  critical  to  continuing  ROI  calculations.  The 
goal  is  to  calculate  an  annual  value  for  particular  training.  This  involves  assigning 
monetary  values  to  training  results  such  as  increased  productivity,  service,  and/or  quality. 
Therefore,  training  value  is  derived  from  the  estimated  annual  value  of  a  training  program 
for  all  program  participants  and  is  called  the  total  program  benefits  (Phillips  1996).  These 
program  benefits  can  also  be  viewed  as  costs:  costs  that  were  avoided  due  to  training. 

To  complete  the  ROI  calculation,  the  cost  of  the  training  must  be 
calculated.  Tabulating  training  program  costs  includes  all  costs  related  to  the  design  and 
delivery  of  the  training.  This  includes  instructor  fees,  various  logistics  fees,  and  the 
opportunity  costs  for  the  employees  who  attend  training.  The  result  of  this  step  is  the  total 
organizational  cost  related  to  conducting  the  training  program. 

The  training  ROI  is  computed  by  dividing  the  estimated  value  of  the 
training  by  the  costs  to  perform  the  training.  Comparing  these  training  ROI  ratios  across 
different  training  programs  provides  insight  into  training  program  efficiency  and 
effectiveness. 

Phillips’  ROI  model  is  easy  to  understand  and  valuable  as  a  tool  to 
manage  the  administration  of  training  programs.  There  is  no  reason  that  it  could  not  be 
applied  to  calculate  the  perfonnance  of  various  graduate  education  programs.  It  provides 
a  useful  to  for  human  resource  decision  makers.  However,  it  fails  to  reveal  how  skills  and 
knowledge  affect  organizational  processes.  This  is  a  significant  distinction — measuring 
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the  efficiency  of  getting  knowledge  into  an  organization  is  different  from  measuring  the 
efficiency  of  using  the  knowledge  to  achieve  organizational  objectives. 

4.  Surrogate  Performance  Measures 

The  last  category  in  this  literature  review  highlights  studies  that  examine  the  value 
of  graduate  education  within  organizations.  The  primary  subjects  in  most  of  these  studies 
are  naval  officers  who  have  graduate  degrees  from  the  Naval  Postgraduate  School  (NPS), 
Monterey,  California.  These  studies  employ  a  variety  of  surrogate  measurements  to 
estimate  the  organizational  impact  of  graduate  education.  In  general,  these  studies 
explore  relationship  between  graduate  education  and  organizational  rewards.  Promotion 
and  retention  data  are  the  most  commonly  used  surrogate  metrics.  Table  3  provides  a 
summary  of  these  studies. 


Study 

Approach 

Discussion 

Relationship  between  on-the- 
job  productivity  and  graduate 
education  (Bowman  and 

Mehay  1999). 

Navy  officers  with  funded 
graduate  degrees.  Data 
associated  with  promotion  to 
0-4. 

On-the-job  productivity  is 
drawn  from  supervisor 
evaluations,  specifically 
performance  report  assessment 
of  “recommended  for  early 
promotion.” 

The  research  indicated  that 
officers  with  graduate  degrees 
were  more  likely  to  be 
promoted.  Graduates  of  NPS 
had  an  increased  likelihood. 

However,  the  research  further 
indicated  that  a  large  part  of 
the  relationship  between 
promotion  and  education  is 
due  to  “unobserved 
attributes.” 

Economic  returns  of  NPS 
education  for  Marine  Corps 
officers  (Branigan  2001). 

Marine  Corps  officers  with 
and  without  NPS  degrees. 

Data  associated  with 
promotion  to  0-5. 

Value  based  on  retention 
statistics  and  promotion  rates 
to  the  grade  of  0-5  for  officers 
with  graduate  degrees. 

The  research  indicates  that 
graduate  education  positively 
affects  retention  and 
promotion. 

Unable  to  isolate  the  attributes 
of  NPS  education  from  non- 
NPS  graduate  education. 

Individual  productivity  related 
to  graduate  education  [Cymrot 
1986  (Branigan  2001,  12-14)]. 

Navy  officers  with  funded 
graduate  degrees. 

Time  in  previous  grade  (time 
to  promotion)  serves  as  an 
indicator  of  productivity — 
faster  promotion  is  tied  to 
increased  productivity. 

The  research  indicates  that 
graduate  education  does  not 
affect  promotion  rate  (the 
productivity  surrogate). 
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Effects  of  graduate  education 
on  productivity  and  job 
performance  (Usan  and  Utoglu 
1999). 

Civilian  Department  of 

Defense  (DoD)  employees 
with  at  least  a  Bachelor’s 
degree. 

Assumes  that  productivity  and 
job  performance  are  reflected 
in  salary,  promotion,  retention, 
and  performance  rating. 

The  research  indicates  that 
graduate  degree  holders  earn 
more,  are  more  likely  to 
promote,  receive  higher 
performance  ratings,  and  are 
less  likely  to  remain  in  the 

DoD. 

Effects  of  graduate  education 
on  measures  of  job 
performance  (Wielsma  1996). 

Cohort  of  Marine  officers 
commissioned  in  1980  with 
and  without  graduate 
education.  Data  associated 
with  promotion  to  0-4. 

Job  performance  is 
approximated  from  an  average 
performance  index.  The 
performance  index  is 
calculated  from  fitness  report 
ratings. 

The  research  indicates  that 
officers  with  graduate 
education  have  a  higher 
average  performance  index. 

However,  average 
performance  index  does  not 
correlate  to  promotion. 

Relationships  between 
academic  achievement  and  job 
performance  (Wise  1975  a; 
1975b). 

College  graduates  in  a  large 
manufacturing  firm. 

Job  performance  is  measured 
by  changes  in  salary  and  grade 
level. 

The  research  indicates  that 
academic  achievement  is  a 
significant  determinant  of 
salary  and  promotion. 

The  research  is  unable  to 
separate  specific  contributions 
of  education  from  preexisting 
abilities. 

Table  3.  Studies  Relating  Education  to  Organizational  Performance 


While  many  of  the  studies  in  Table  3  use  the  term  productivity,  none  provide  an 
explicit  definition.  Most  address  the  difficulty  of  providing  a  precise  definition  and 
explicitly  assert  that  productivity  is  best  measured  by  what  an  organization  rewards.  This 
assertion  assumes  that  organizational  rewards  materialize  in  the  form  of  promotions, 
retention,  positive  performance  ratings,  and  increased  salary  (in  private  organizations). 
Recognizing  a  positive  correlation  between  education  and  organizational  rewards 
validates  the  basic  assumption  that  an  educated  employee  is  a  better  employee.  However, 
these  studies  do  not  help  decision  makers  tease  out  direct,  unambiguous  contributions  of 
graduate  education  to  organizational  processes. 
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The  ability  to  estimate  accurately  and  reliably  the  return  on  an  educational 
investment  remains  an  elusive  problem.  While  this  literature  review  reveals  many 
approaches  to  measure  the  knowledge  as  an  organizational  capability,  they  fail  to  show 
how  the  use  of  these  capabilities — graduate  education  in  this  case — directly  affects  core 
organizational  processes.  The  number  and  variety  of  research  approaches  presented  in 
this  review  highlight  the  far-reaching  desire  to  gain  additional  insight  into  the  dynamics 
between  education  and  performance.  To  move  beyond  these  approaches  and  evaluate  the 
contribution  of  knowledge  within  an  organization,  two  challenges  must  be  resolved:  an 
organization’s  graduate  education  objectives  must  be  identified  (how  is  it  intended  to  be 
used)  and  actions  that  contribute  to  these  objectives  must  be  collected  and  assigned  value 
in  common  units. 

B.  MARINE  CORPS  GRADUATE  EDUCATION  PROCESSES 

This  proof  of  concept  deals  with  the  value  of  graduate  level  education  to  the 

United  States  Marine  Corps.  The  majority  of  the  Marine  Corps’  fully  funded  graduate 
education  is  managed  under  the  Special  Education  Program  (SEP).  The  SEP  seeks  to 
align  graduate  education  curriculums  with  the  education  requirements  of  specific  billets 
or  jobs.  This  curriculum-billet  relationship  tacitly  demonstrates  that  the  Marine  Corps 
expects  a  specific  impact  to  its  core  processes  because  of  graduate  education. 

A  procedure  to  estimate  the  return  on  educational  investment  is  very  appropriate 
to  the  Marine  Corps  SEP  for  two  key  reasons.  First,  graduate  education  is  expensive. 
Perhaps  the  greatest  organizational  expense  is  the  time  that  an  officer  spends  away  from 
the  organization  during  education.  However,  the  Corps  continues  to  pay  this  expense  with 
the  expectation  that  the  education  will  further  enhance  the  organization.  This  prompts  the 
second  reason — without  a  means  to  estimate  the  return  on  educational  investments,  the 
Corps  is  unable  to  detennine  if  its  finite  educational  investments  are  being  employed  to 
greatest  organizational  advantage. 

Responding  to  the  first  challenge  could  be  solved  in  part  by  using  a  human 
resource  approach  to  training  return  on  investment  such  as  Phillips’.  This  could  help 
reveal  how  effectively  Marine  officers  are  being  educated.  But  this  approach  fails  to 
respond  to  the  second  challenge.  Analyzing  what,  how,  to  what  extent  knowledge 
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investments  contribute  to  process  outputs  requires  a  different  approach.  In  other  words, 
the  approach  must  extend  beyond  managing  how  effectively  knowledge  is  acquired  and 
provide  the  tools  to  manage  where  and  how  knowledge  makes  the  greatest  impact. 

In  order  to  understand  the  implications  of  developing  approaches  to  resolve  these 
challenges,  the  SEP  is  described  in  this  section. 

The  Marine  Corps’  SEP  is  chartered  to  manage  the  majority  of  the  Marine  Corps’ 
graduate  level  education  requirements  and  officer  assignments.  This  program  includes  the 
validation  and  assignment  of  officers  to  roughly  four  hundred  SEP  billets.  This  section 
consists  of  a  review  of  the  SEP  in  terms  of  stakeholders,  processes,  and  challenges.  It  is 
intended  to  present  the  current  state  of  the  Marine  Corps  graduate  education  environment. 
This  background  is  important  to  understanding  subsequent  data  collection,  analysis,  and 
rec  ommendations . 

1.  Special  Education  Program  Stakeholders 

There  are  several  stakeholders  in  the  Marine  Corps’  SEP  processes.  In  short,  they 
are  decision  makers  who  perfonn  specific  functions  to  identify,  validate,  design,  and/or 
evaluate  graduate  education  programs. 

•  Education  Providers  are  the  accredited  academic  institutions  (military  and 
civilian)  that  provide  graduate  degrees  to  military  officers. 

•  Manpower  Management  and  Officer  Assignments  (MMOA)  is  the  Marine 
Corps  activity  that  assigns  officers  to  billets.  There  is  an  officer  in  MMOA 
responsible  for  the  assignment  of  SEP  officers.  This  officer  is  commonly 
called  the  SEP  Monitor. 

•  Owning  Commands  are  commands  that  are  authorized  to  maintain  SEP 
billets.  The  number  of  SEP  billets  is  fixed.  For  a  command  to  gain  a  SEP 
billet,  a  command  must  relinquish  an  existing  billet.  There  are  seventy- 
eight  owning  commands  in  the  SEP  (Marine  Corps  2000). 

•  Program  Sponsors  are  military  activities  that  sponsor  graduate 
curriculums.  They  serve  as  the  liaison  between  the  military  service  and 
academic  institutions.  The  Program  Sponsors  work  with  educational 
institutions  to  ensure  that  military  requirements  of  graduate  education  are 
achieved.  Program  sponsors  are  usually  Occupation  Field  (OCCFLD) 
Managers.  These  Marine  Corps  officers  manage  collections  of  similar 
Military  Occupation  Specialties  (MOS).  Twenty-six  different  MOSs  are 
authorized  at  the  completion  of  graduate  education  and  carry  a  96XX 
code.  (Marine  Corps  1993). 
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•  Special  Education  Program  Officers  are  officers  who  have  applied  and 
been  selected  to  participate  in  the  SEP.  Acceptance  into  the  SEP  incurs  a 
tour  in  a  SEP  payback  billet  and  a  four-year  active  duty  commitment  after 
graduation.  The  SEP  process  is  designed  to  ensure  that  SEP  officers  are 
assigned  to  a  SEP  billet  that  corresponds  to  their  education  at  the 
completion  of  their  education.  Some  SEP  officers  serve  repeated  tours  in 
SEP  billets,  but  this  is  not  a  requirement  of  the  program. 

•  Total  Force  Structure  Division  (TFSD)  is  the  Marine  Corps  activity  that 
manages  the  Corps’  Table  of  Organization  and  Equipment.  They  are  the 
approving  authority  for  all  personnel  billets,  including  SEP  billets. 

2.  Special  Education  Program  Processes 

There  are  several  processes  related  to  managing  the  Marine  Corps’  SEP.  The 
stakeholders  work  together  to  accomplish  these  activities.  In  this  paper,  they  are  grouped 
along  three  functions  to  simplify  their  description:  requirements,  provisioning,  and 
evaluation. 


a.  Requirements 

The  SEP  process  to  identify  graduate  level  education  requirements  is 
depicted  in  Figure  4.  This  figure  shows  the  flow  of  actions  to  identify  and  validate 
Marine  Corps  billets  that  require  graduate  education. 


Graduate 
education  need 


Owning 

command 

*  determines  ^  Initiate  BEEC  — ► 

OCCFLD 

manager 

^  TFSD 

review 

graduate 

education 

review 

requirement 

i 

i 

Invalidated 

Unavailable  force 

requirement 

structure 

Validated  SEP 
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Figure  4.  Identification  and  Validation  of  Graduate  Education  Requirements 


There  are  two  main  parts  in  this  requirements  process.  Billet  Education 
Evaluation  Certificates  (BEEC)  are  used  to  describe  the  graduate  education  requirements 
for  specific  billets.  These  certificates  are  initiated  by  owning  commands  to  articulate 
graduate  level  KSAs  required  in  a  specific  billet.  The  education  requirements  are 
reviewed  and  validated  by  appropriate  OCCFLD  Managers.  The  TFSD  performs  the  final 
approval  for  BEECs. 
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The  Marine  Corps  Total  Force  Structure  Process  (TFSP)  provides 
guidelines  for  managing  the  United  States  Marine  Corps’  manpower  billets.  This  process 
designates  TFSD  as  the  manager  of  the  Marine  Corps’  force  structure.  The  TFSP 
prescribes  how  a  command  may  request  a  change  to  an  existing  billet.  The  procedures  to 
initiate  special  education  requests  using  BEECs  (described  above)  are  explained  in  the 
TFSP  (Marine  Corps  1999a).  Total  Force  Structure  Division  approval  of  a  BEEC  is 
required  to  authorize  a  SEP  billet  for  a  specific  owning  command. 

b.  Provisioning 

The  processes  to  identify  SEP  officers  and  manage  academic  programs  are 
depicted  in  Figure  5.  This  figures  shows  the  relationships  between  SEP  stakeholders  to 
accomplish  the  three  main  element  of  provisioning:  requirements  dissemination,  student 
selection,  and  curriculum  review.  These  responsibilities  are  shared  between  the  military 
and  education  institutions. 

Educational  Skill  Requirements  (ESR)  are  the  essential  elements  of  an 
academic  program  defined  by  the  Program  Sponsor.  An  ESR  is  provided  for  each  SEP 
curriculum  and  formally  describes  the  education  requirements  from  a  military  service 
perspective.  Education  institutions  use  the  ESR  to  design  academic  programs  to  meet 
service  education  requirements.  Education  Skill  Requirement  documents  may  specify 
specific  curriculum  requirements  in  addition  to  describing  the  general  intent  of  the 
education  in  terms  of  post-education  application  environments. 
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Figure  5.  Selecting  Special  Educaiton  Program  Officers  and  Managing  Academic 

Requirements 

To  maintain  a  flow  of  officers  in  the  SEP  pipeline,  the  Marine  Corps 
solicits  applicants  for  future-year  SEP  education  opportunities.  The  appropriate  education 
institution  first  reviews  the  potential  SEP  officer’s  application  to  make  an  academic 
acceptance  decision.  Applications  that  receive  tentative  academic  acceptance  are 
reviewed  in  a  board  process  managed  by  MMOA.  Applications  that  pass  service 
requirements  are  assigned  to  SEP  curricula  based  on  the  applicants’  desires  and  the  needs 
of  the  Marine  Corps.  (Marine  Corps  2002) 


The  third  part  of  the  process  consists  of  curriculum  monitoring. 
Educational  institutions  conduct  formal  curriculum  reviews  on  a  recurring  schedule.  This 
review  process  involves  the  Program  Sponsors  and  is  geared  towards  aligning  curriculum 
to  ESRs.  These  reviews  assess  curriculum  design  and  execution  in  terms  of  military 
service  needs.  Additionally,  they  evaluate  curriculum  resources  and  research  efforts  (NPS 
2002a). 
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c.  Evaluation 

The  Marine  Corps  process  to  evaluate  the  effectiveness  of  its  SEP  billet 
distribution  is  depicted  in  Figure  6.  The  process  to  evaluate  the  overall  effectiveness  of 
the  SEP  is  the  SEP  billet  review.  The  review  is  designed  to  optimize  the  distribution  of 
SEP  billets  throughout  the  Marine  Corps.  The  review  has  two  specific  objectives: 
identify  the  proper  distribution  of  SEP  billets  across  owning  commands  and  identify  the 
correct  mix  of  graduate  education  disciplines.  Stakeholders  in  the  review  process  are 
MMOA,  OCCFLD  Managers,  TFSD,  and  SEP  billet  owning  commands. 


Owning  command 
submits  education 
requirement 

i 

Billet  retained  in 
SEP 


SEP  billet 
review 


Billet  removed 
from  SEP 


Figure  6.  Special  Education  Program  Billet  Evaluation  Process 


This  review  process  relies  heavily  on  billet  information  provided  in 
BEEC,  as  well  as  the  judgment  of  the  members  conducting  the  review.  The  most  recent 
SEP  billet  review  stalled  after  two  years  of  effort  without  substantial  changes  to  the  pre¬ 
existing  SEP  billet  structure  (Klinger  2002a;  Marine  Corps  2000,  2001,  2002). 
Completing  this  review  process  to  adjust  resource  allocations  is  complicated  by  the  wide 
breadth  and  diversity  of  SEP  billets  and  owning  commands.  Furthermore,  the  reallocation 
decisions  are  based,  in  large  part,  on  the  limited  information  contained  in  the  BEECs. 


By  its  design  and  charter,  this  review  process  does  not  try  to  detennine  the 
demonstrated  value  of  graduate  education  to  Marine  Corps  processes.  The  process  is 
solely  based  on  pairing  validated  education  requirements  to  specific  owning  commands. 
In  taking  this  approach,  The  Marine  Corps  appears  to  assume  that  its  investments  in 
graduate  education  are  providing  value  to  the  organization  and  that  the  Corps  does  not 
need  to  expend  resources  to  quantify  this  supposed  value.  The  SEP  billet  review  is  a 
requirements-based  process  that  minimally  considers  feedback  from  actual  graduate 
education  application  of  SEP  officers  assigned  to  SEP  billets. 
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3.  Challenges  to  Assessing  Graduate  Education  Value 

The  SEP  processes  to  identify  graduate  education  needs,  influence  educational 

processes,  and  evaluate  SEP  officer  perfonnance  are  generally  effective  but  not  managed 
to  attain  their  fullest  potential.  High  priority  graduate  education  needs  are  identified  and 
satisfied  with  the  assignment  of  officers  who  have  accredited  education.  The  value  of 
SEP  officers  to  owning  commands  is  demonstrated  in  part  by  the  competition  between 
commands  to  acquire  and/or  maintain  SEP  billets.  The  current  processes  have  evolved  to 
meet  many  of  the  Corps’  graduate  education  requirements,  but  there  remain  shortcomings 
to  these  processes. 

Many  of  these  shortcomings  are  highlighted  by  the  recent  SEP  billet  review.  The 
review  consumed  many  resources  and  did  not  result  in  substantial  realignment  of  SEP 
billets.  This  could  indicate  that  the  pre -review  billet  alignment  was  appropriate,  or 
perhaps  it  signals  that  the  reviewers  lacked  the  appropriate  tools  to  make  more  effective 
graduate  education  allocations.  Furthermore,  the  billet  review  process  is  based  on  the 
documentation  of  requirements  as  opposed  to  demonstrated  performance. 

A  Marine  Corps  process  that  links  the  skills  and  knowledge  acquired  during 
graduate  education  to  improvements  to  and/or  efficiencies  in  core  processes  does  not 
exist.  This  capability  would  improve  the  Marine  Corps’  ability  to  effectively  manage  its 
limited  population  of  officers  with  organizationally  funded  graduate  education.  The 
review  of  current  Marine  Corps  SEP  processes  reveals  several  difficulties  in  the 
organization’s  ability  to  detennine  the  value  of  its  graduate  education  investments. 

a.  Consistent  Measurement 

The  Marine  Corps  Performance  Evaluation  System  (PES)  is  used  to 
evaluate  billet  performance  of  Marines  with  rank  from  sergeant  to  general  (Marine  Corps 
1998).  As  a  single  personnel  evaluation  system,  the  PES  serves  the  Corps  well.  Because 
it  is  optimized  to  assess  the  general  performance  attributes  of  a  wide  range  of  Marines, 
the  system  does  not  ensure  specific  and  standard  insights  into  the  impact  of  graduate 
education  on  billet  perfonnance.  The  system’s  inability  to  document  the  influence  of 
graduate  education  is  further  limited  by  the  inconsistencies  of  individual  supervisor 
evaluations.  Therefore,  there  is  no  specific  process  to  assess  the  performance  of  officers 
in  SEP  billets. 
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b.  Billet  Lock-in 

The  process  to  prioritize  SEP  billets  to  achieve  the  greatest  organizational 
value  to  the  Marine  Corps  lacks  responsiveness.  Once  a  billet  is  designated  with  a  SEP 
requirement,  the  removal  of  that  designation  is  difficult.  Because  there  are  a  finite 
number  (about  four  hundred)  of  SEP  billets,  creating  a  new  SEP  billet  requires  the 
surrender  of  another.  This  fixed  number  of  SEP  billets  often  results  in  a  battle  between 
billet  owners  who  seek  to  maintain  their  SEP  graduates. 

This  battle  would  appear  to  validate  the  value  of  SEP  officers  to  a 
command.  However,  the  desirability  of  SEP  billets  is  not  solely  based  on  the  SEP 
officer’s  capabilities.  Billets  that  require  SEP  officers  are  filled  at  a  higher  rate  than  many 
non-SEP  billets.  In  other  words,  SEP  billets  ensure  a  higher  probability  of  actually  having 
an  officer  to  fill  a  billet,  regardless  of  whether  the  SEP  officer’s  education  is  employed  or 
not.  The  only  process  to  prioritize  SEP  investments  across  the  Marine  Corps  is  a 
complete  review  of  all  SEP  billets,  and  the  various  challenges  related  to  this  process  were 
highlighted  above. 

c.  Curriculum  Feedback 

The  current  processes  used  to  manage  the  SEP  have  no  direct  Marine 
Corps  feedback  to  post-graduate  curriculum  managers.  This  causes  a  lag  in  the  SEP 
curriculums  to  meet  dynamic  graduate  education  needs.  Additionally,  it  complicates 
processes  that  attempt  to  relate  SEP  billet  performance  to  curriculum  content.  The 
absence  of  this  feedback  mechanism  makes  the  use  of  learning  evaluation  methods  more 
difficult  and  inhibits  the  ability  of  decisions  makers  to  optimize  educational  investment. 

C.  RESEARCH  QUESTIONS 

The  review  of  the  relevant  literature  concerning  the  organization  value  of 
graduate  level  education  reveals  a  variety  of  approaches  to  estimate  the  value  of 
education  within  an  organization.  The  existing  body  of  research  supports  the  effort  to 
establish  methods  to  connect  investments  in  graduate  education  to  organizational 
processes.  Components  of  the  examined  studies  take  practical  form  in  some  of  the  Marine 
Corps’  current  management  of  graduate  education  programs.  Based  on  the  original  thesis 
and  results  of  this  literature  review,  the  following  research  questions  precipitate. 
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How  well  does  the  NPS  Information  System  Technology  (1ST)  curriculum 
meet  Marine  Corps  graduate  level  Information  Technology  (IT)  education 
requirements? 

What  is  the  value  of  the  1ST  education  to  the  Marine  Corps? 

How  does  the  NPS  1ST  curriculum  improve  productivity  within  Marine 
Corps  SEP  billets? 

What  effect  does  personal  desire  have  on  education  value? 

Is  there  a  relationship  between  social  connections  formed  during  graduate 
education  and  subsequent  productivity? 

Can  a  Knowledge  Value  Added  method  be  used  to  assess  the  current  value 
of  graduate  education? 
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II.  DATA  COLLECTION 


This  research  provides  a  proof  of  concept  for  a  method  to  estimate  the  value  of 
graduate  education.  This  chapter  explains  the  methods  implemented  to  gather  the  data 
used  to  develop  this  concept.  The  objective,  scope,  and  instruments  of  data  collection  are 
addressed. 

A.  OBJECTIVE 

Data  collection  focuses  on  gathering  infonnation  appropriate  to  answer  the 
research  questions.  These  research  questions  are  tied  to  detennining  the  value  of  graduate 
education.  In  this  study,  the  value  of  graduate  education  is  assessed  based  on  the  impact 
of  education  to  core  organizational  processes.  The  collected  data  collection  is  central  to 
establishing  connections  between  education  use  and  changes  in  process  output.  Once 
established,  these  connections  reveal  what  elements  of  graduate  education  are  being  used 
and  what  impact  they  generate. 

B.  SCOPE 

Graduate  education  program,  subject  population,  and  education  application  define 
the  scope  of  this  investigation.  The  Information  System  Technology  (1ST)  program  at  the 
Naval  Postgraduate  School  (NPS)  is  examined.  The  subject  population  of  this  study 
consists  of  Marine  Corps  officers  who  have  participated  in  the  1ST  program.  Lastly,  the 
application  of  the  1ST  education  is  limited  to  Marine  Corps  billets  designated  as  requiring 
this  education. 

1.  Information  System  Technology 

The  Naval  Postgraduate  School  is  the  Navy  and  Marine  Corps’  corporate 
institution  for  graduate  level  technical  education.  The  NPS  offers  an  accredited  Masters 
of  Science  in  Infonnation  Technology  Management  (ITM)  as  part  of  its  1ST  curriculum. 

This  curriculum  provides  officers  with  the  knowledge  of  information 
systems  technology  to  include  computer  and  telecommunications  systems, 
software  engineering,  networked  and  distributed  applications,  database 
management  systems,  and  decision  support  systems  in  the  military 
services.  Students  .  .  .  gain  proficiency  in  infonnation  systems,  economics 
and  management  necessary  for  the  critical  management  decisions  needed 
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in  the  development  and  utilization  of  complex  and  evolving  computer- 
based  military  systems  (NPS  2002c). 

This  curriculum  is  designed  to  provide  the  education  described  by  the  various  1ST 
program  sponsors.  In  practice,  these  skill  sets  usually  exceed  the  scope  of  typical 
accredited  graduate  degree  programs.  To  accommodate  these  additional  skill 
requirements,  the  NPS  supports  curriculums  that  often  exceed  standard  degree 
requirements  in  order  to  meet  the  totality  of  sponsor  objectives.  This  responsiveness  to 
sponsor  requirements  is  the  raison  d’etre  of  the  NPS. 

The  distinction  between  the  1ST  curriculum  and  the  ITM  degree  is  worth 
additional  description  to  clarify  the  different  objectives  of  the  education.  Successful 
completion  of  the  NPS  1ST  curriculum — the  education  that  the  program  sponsor  wants — 
provides  an  officer  with  the  Marine  Corps  Military  Occupational  Specialty  (MOS)  code 
of  9648,  Management,  Data  Systems  Officer  (Marine  Corps  1993).  On  the  other  hand, 
the  ITM  degree — included  as  part  of  the  1ST  curriculum  and  particularly  valued  by  the 
student — represents  only  a  part  of  the  1ST  curriculum.  In  fact,  the  ITM  degree  requires 
only  fifty-two  credit  hours  of  study  and  forms  the  nucleus  of  the  125  credit  hours 
required  by  the  1ST  curriculum  (Cook  2002a). 

The  NPS  ITM  degree  predates  the  current  1ST  curriculum.  The  current  1ST 
curriculum  evolved  from  previous  curriculums  to  adapt  to  the  changing  program  sponsor 
needs.  The  1ST  curriculum  most  correctly  describes  the  educational  requirements  of  the 
Marine  Corps’  program  sponsor;  however,  the  Marine  Corps  Special  Education  Program 
(SEP)  commonly  refers  to  the  program  as  ITM.  This  is  based  on  the  foundational  degree 
that  predates  the  9648  MOS  producing  curriculum. 

a.  Program  Sponsor 

The  Program  Sponsor  influences  the  composition  of  the  1ST  curriculum  by 
describing  education  requirements.  The  Marine  Corps’  Program  Sponsor  for  the  1ST 
curriculum  is  the  Director,  Command,  Control,  Communications,  and  Computers, 
Headquarters  Marine  Corps.  The  1ST  curriculum  also  has  a  Navy  Program  Sponsor, 
Director,  Space,  Infonnation  Warfare,  Command  and  Control  Division,  Navy  Staff.  The 
Navy  sponsor  provides  the  bulk  of  the  financial  support  related  to  the  development  and 
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management  of  the  1ST  curriculum  and  subsequently  sets  the  educational  requirements 
for  the  1ST  curriculum.  The  Marine  Corps  sponsor  has  traditionally  supported  and 
accepted  the  Navy  educational  requirements  and  does  not  take  an  active  role  in  affecting 
the  overall  design  of  the  1ST  curriculum  except  where  specific  Marine  Corps  issues 
warrant  differences. 

To  influence  curriculum,  the  Navy  Program  Sponsor  disseminates  the 
Educational  Skill  Requirements  (ESR)  for  the  1ST  curriculum.  The  ESR  articulates  the 
Knowledge  Skill,  and  Abilities  (KSA)  objectives  of  the  curriculum.  The  1ST  ESR  states 
that  graduates  should  have  the  KSAs  to  engineer  and  manage  information  systems  afloat 
and  ashore.  To  realize  the  KSA  goals,  several  areas  are  emphasized  (NPS  2002b). 

•  Strategy  and  Policy — the  ability  to  think  strategically  and  discern  the 
relationship  between  political  interests  and  military  application. 

•  Space,  Information  Warfare,  and  Command  and  Control  Professional 
Practices 

•  Software  Development — the  ability  to  manage  software  programs. 

•  Information  System  Technology 

•  Information  System  Analysis  and  Management 

•  Military  Application  of  analytical  methods,  technical  expertise,  and 
operational  experience. 

•  Independent  Research  in  the  form  of  a  thesis. 

It  is  important  to  note  that  while  development  of  the  1ST  ESR  is  the 
responsibility  of  the  Program  Sponsor,  the  NPS  takes  an  active  role  in  advocating 
curriculum  initiatives.  The  provider-based  initiatives  are  negotiated  with  the  Sponsor  and 
often  lead  changes  in  the  ESR  and  curriculum  (Cook  2002a). 

This  ESR  document  constructs  the  foundation  of  the  NPS  1ST  curriculum. 
Because  the  Marine  Crops  has  some  specific  educational  requirements  not  resident  in  the 
Navy  ESR,  the  1ST  curriculum  for  Marine  students  contains  subtle  differences.  Adding  or 
substituting  existing  NPS  courses  into  the  Marine  specific  curriculum  resolves  these 
minor  differences. 
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b.  Marine  Corps  Requirements 

While  the  ESR  process  is  used  to  articulate  educational  requirement  to 
academic  institutions,  the  only  pseudo-formal  method  to  identify  educational 
requirements  within  the  Marine  Corps  is  through  the  collection  and  validation  of  Billet 
Education  Evaluation  Certificates  (BEEC).  As  explained  previously,  the  Marine  Corps 
generally  accepts  the  1ST  ESR  that  the  Navy  Sponsor  develops.  However,  when  asked 
how  the  Marine  Corps  could  compile  and  articulate  specific  education  requirements  for 
the  9648  MOS,  the  9648  Occupation  Field  (OCCFLD)  Manager  stated  that  he  would 
have  to  query  the  existing  9648  MOS  billet  holders  (Klinger  2002a).  This  would  probably 
be  accomplished  using  the  existing  BEEC  format. 

Billet  Education  Evaluation  Certificates  are  designed  to  explain  and  justify 
a  SEP  billet’s  educational  requirements;  see  fonnat  in  Appendix  A.  A  BEEC  is  required 
and  associated  with  each  SEP  billet  (Marine  Corps  1999a).  Marine  Corps  Curriculum 
Sponsor  maintains  a  BEEC  for  each  SEP  MOS.  The  BEEC  describes  what  subjects  a 
curriculum  should  include.  It  also  includes  the  educational  justification  for  specific  billet 
requirements. 

By  definition,  BEECs  for  9648  MOS  SEP  billets  document  the  graduate 
1ST  education  requirements  of  the  Marine  Corps.  Collecting  and  analyzing  the  current 
9648  MOS  BEECs  serves  two  purposes.  First,  the  graduate  education  justification  of  each 
BEEC  should  describe  a  set  of  core  processes  that  the  billet  holder  will  perform.  Second, 
BEECs  define  the  curricular  components  required  to  effectively  function  in  the  billets’ 
core  processes. 

The  9648  MOS  OCCFLD  Manager,  the  owning  commands  of  9648  MOS 
billets,  and  the  Total  Force  Structure  Division,  Marine  Corps  Combat  Development 
Command,  maintain  BEECs  for  the  9648  MOS  SEP  billets.  Existing  file  copies  of  the 
9648  MOS  BEECs  are  requested  as  part  of  this  study’s  data  collection  to  define  the 
Marine  Corps  specific  education  requirements  for  its  9648  MOS  SEP  billets. 

c.  Information  System  Technology  Curriculum 

The  NPS  curriculum  crafted  to  meet  the  1ST  ESRs  is  designed  as  an  eight- 
quarter/two-year  program.  Within  the  general  subject  areas,  students  may  take  elective 
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courses  to  specialize  their  studies  in  fields  such  as  networks  and  security.  The  curriculum 
also  allows  students  to  pursue  dual  degrees  in  related  fields;  however,  dual  degrees  are 
usually  restricted  to  students  possessing  strong  undergraduate  or  graduate  background  in 
parallel  disciplines.  The  default  1ST  curriculum  for  Marine  students  is  presented  in 
Appendix  B. 

d.  Information  System  Technology  Curriculum  Components 

In  this  study,  the  1ST  curriculum  is  divided  in  two  ways  to  assist  in 
effective  data  collection.  First,  the  results  of  the  1ST  curriculum — learning — are  viewed 
as  providing  the  student  with  practical  hands-on  information  technology  skills,  as  well  as, 
theoretical  or  conceptual  knowledge  relating  to  information  technology.  An  explicit 
description  of  this  division,  as  it  relates  to  1ST  courses,  is  not  discussed.  However,  this 
distinction  will  be  highlighted  during  data  collection  and  used  to  assess  the  relative 
weighting  between  skills  and  knowledge  derived  from  the  1ST  curriculum. 

The  second  division  of  the  1ST  curriculum  breaks  the  courses  into  fourteen 
topic  areas.  Table  4  lists  the  topics  and  associates  them  with  the  default  1ST  list  of 
courses  presented  in  Appendix  A.  The  table  also  shows  the  associated  course  hours 
dedicated  to  each  1ST  topic.  The  1ST  curriculum  allows  for  elective  study  beyond  the 
courses  listed  in  Table  4,  and  these  additional  courses  are  associated  with  topics  as  they 
are  revealed  in  data  collection.  These  fourteen  topic  areas  have  been  defined  by  the 
author  and  are  consistent  with  the  School’s  Department  of  Information  Sciences 
Associate  Chairman  for  Operations  (Cook  2002a). 
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Topic 

Courses 

Hours 

Acquisition/  Program 
Management 

Principles  of  Systems  Acquisition  and  Program 
Management 

6 

Command  and  Control 

Introduction  to  Command,  Control,  Communication, 
Computer  and  Intelligence  Systems  in  DoD 

Principles  of  Information  Operations 

9 

Communication  System 
Engineering 

Computer  Architecture  and  Operating  Systems 

Introduction  to  Communications  Systems  Engineering 
for  ITM 

Introduction  to  Communications  Systems  Engineering  II 

Electronic  Communications  Systems  for  ITM 

22 

Computer  Networking 

Computer  Networks:  Wide  Area/Local  Area 

Process  Re-Engineering  with  Information  Technology 

C4ISR  Systems 

14 

Computer/  Network 
Security 

Information  Assurance:  Introduction  to  Computer 

Security 

6 

Database 

Fundamentals  of  Database  Technology 

5 

Decision  Support 

Systems 

Fundamentals  of  Decision  Support  Systems 

5 

Economic/  Financial 
Management 

C4ISR  System  Evaluation 

Principles  of  Information  Systems  Evaluation 

Financial  Management  in  the  Armed  Forces 

10 

Managing  Change 

Managing  Planned  Change  in  Complex  Organizations 

Information  Systems  Management 

8 

Professional  Military 
Education 

Strategy  and  Policy:  The  American  Experience 

6 

Software  Engineering 

Software  Design 

Software  Engineering  and  Management 

10 

Software  Language 
Programming 

Introduction  to  Object  Oriented,  Event-Driven 
Programming  Using  Microsoft  Visual  Basic 

5 

Space  Operations 

Space  Technology  and  Applications 

Military  Satellite  Communications 

6 

Statistical  Analysis 

Statistics  for  Technical  Management 

Operations  Research  for  Computer  Systems  Managers 

10 

Table  4.  Information  System  Technolgy  Curriculum  Topics 
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2.  Subject  Population 

The  subjects  for  this  data  collection  are  Marine  officers  who  have  the  9648  MOS. 
These  officers  have  generally  completed  between  four  and  twelve  years  of  active  duty 
service  before  attending  the  NPS.  They  volunteered  for  and  were  accepted  into 
organizationally  funded  education  in  return  for  a  commitment  to  apply  that  education  in  a 
SEP  “payback”  billet.  They  were  assigned  the  9648  MOS  upon  completion  of  their  NPS 
curriculums.4 

The  population  of  SEP  graduates  used  in  this  study  is  drawn  from  the  Marine 
Corps  Manpower  Management  and  Officer  Assignments  database.  The  SEP  Monitor 
maintains  a  list  of  9648  MOS  SEP  officers  dating  bake  to  1987  (Esparza  2002a).  The  list 
contains  122  officers.  Current  contact  information  is  available  for  eighty-four  of  these 
officers.5  These  eighty-four  officers  form  the  pool  of  graduates  surveyed  in  this  research. 

To  assist  in  data  collection  and  analysis,  the  population  is  divided  variously. 
Collection  instruments  allow  the  subject  population  to  be  sub-divided  by  the  five 
categories  described  in  Table  5. 


4  Currently,  the  9648  MOS  is  granted  on  completion  of  the  NPS  1ST  curriculum.  Before  1998,  the  1ST 
curriculum  was  simply  called  ITM.  The  ITM  curriculum  and  degree  were  developed  from  a  merger  of 
Computer  Systems  Management  and  Telecommunications  Systems  Management  in  1990. 

5  The  Navy/Marine  Corps  White  Pages  Directory,  http://sdiego.dir.navy.mil/basic  search  frameset.htm, 
was  used  to  cross  reference  9648  MOS  officers  with  current  e-mail  addresses.  Comprehensive  look-ups 
were  conducted  in  August  2002  and  September  2002. 
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Category 

Domain 

Current  Rank 

Captain 

Major 

Lieutenant  Colonel 

Colonel 

SEP  Billet  Experience 

Currently  in  first  SEP  billet 

Currently  in  second  SEP  billet 

Currently  in  a  post- SEP  billet 

SEP  payback  deferred  until  after  current  billet 

Never  served  in  a  SEP  billet 

SEP  Billet  Dates 

Start  and  End  Date  of  most  recent  SEP  Tour 

NPS  Graduation  Date 

NPS  Graduation  Date 

Education  Specialty 

No  Specialization 

Decision  Support 

Modeling  and  Simulation 

Networks 

Security 

Software  Engineering 

Other 

Table  5. 

Subject  Population  Categories 

3.  Education  Application 

Special  Education  Program  officers  are  normally  assigned  to  SEP  utilization 
billets  upon  completion  of  their  education.  The  Marine  Corps  has  fifty-five  billets 
currently  certified  for  officers  who  have  the  9648  MOS  (Klinger  2002b).  On  rare 
occasions,  graduates  will  not  be  assigned  to  one  of  these  SEP  billets  due  to  manpower 
management  considerations.  Similar  considerations  may  result  in  an  officer  deferring  his 
SEP  billet  tour  until  after  completing  a  non-SEP  billet.  Additionally,  some  officers  may 
serve  repeatedly  in  SEP  billets.  These  9648  MOS  SEP  billets  tend  to  reside  in  the  support 
agencies  of  the  Marine  Corps. 

To  delineate  the  central  functions  of  these  support  agencies,  the  Marine  Corps  has 
specified  several  processes  that  are  critical  to  the  effective  operation  of  the  overall 
organization.  These  nine  processes  represent  those  specific  organizational  functions  that 
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provide  unique,  value  added  benefits  to  an  organization’s  customers — in  this  case,  the 
warfighting  entities  of  the  Marine  Corps.  Table  6  presents  these  nine  processes  and 
identifies  the  process  owners. 

The  fifty-five  9648  MOS  SEP  billets  can  be  associated  with  the  process  owners 
listed  in  Table  6.  This  table  summarizes  the  distribution  of  9648  MOS  SEP  billets  by 
their  affiliation  with  the  nine  process  owners.  The  detailed  list  of  9648  MOS  SEP  billets 
and  specific  process  owner  affiliation  is  presented  in  Appendix  C.  Within  these  nine 
Marine  Corps  processes,  officers  with  the  9648  MOS  generally  support  Infonnation 
Management  efforts.  In  this  context,  the  NPS  1ST  ESR  broadly  defines  information 
management — to  engineer  and  manage  infonnation  systems  afloat  and  ashore.  Table  6  is 
intended  to  show  how  the  Corps’  educational  investments  in  information  management — 
principally,  graduate  education  to  produce  officers  with  the  9648  MOS — are  distributed 
across  the  Marine  Corps  agencies  tasked  to  oversee  the  Corps’  nine  principle  processes. 
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Marine  Corps 
Process 

Description 

Owner 

9648 

Billets 

Acquisition 

Equip  operating  forces  to  accomplish 
warfighting  mission 

Marine  Coips  Systems 
Command  (MCSC) 

17 

Training 

Development 

Develop,  coordinate,  resource,  execute, 
and  evaluate  training  and  education 
concepts,  policies,  plans,  and  programs 

Training  and 

Education  Command 
(TECOM) 

10 

Human  Resource 
Development 

Plan,  execute,  and  coordinate  personnel 
policies 

Manpower  and 

Reserve  Affairs 
(M&RA) 

8 

Resource 

Allocation 

Financial  policy,  programs,  and 
procedures 

Programs  and 

Resources  (P&R) 

5 

Service 

Advocacy 

Marine  Coips  participation  in  the  Joint 
Strategic  Planning  System 

Plans,  Policy,  and 
Operation  (PP&O) 

5 

Total  Force 
Structure 

Allocate  manpower  and  equipment  to 
accomplish  wartime  mission 

Marine  Corps  Combat 
Development 

Command  (MCCDC) 

5 

Information 

Management 

Plan,  direct,  coordinate,  and  provide 
oversight  for  all  command,  control, 
communications,  and  computers 

Command,  Control, 
Communications,  and 
Computers  (C4) 

3 

Infrastructure 

management 

Logistics  policy  and  management 

Installations  and 
Logistics  (I&L) 

1 

Material  Life 

Cycle 

Management 

Ensure  materiel  readiness  to  the 
operating  forces 

Materiel  Command 
(MATCOM) 

1 

Table  6.  Marine  Corps  Processes  (After:  MCCDC  2002  and  After:  Klinger  2002b) 


Table  6  provides  a  convenient  means  to  group  9648  MOS  SEP  billets  based  on 
the  Marine  Corps  allocation  of  these  billets  across  important  organizational  agencies. 
Within  this  framework,  data  collection  seeks  to  establish  the  impact  of  the  NPS  1ST 
education  on  specific  core  processes  within  each  of  these  nine  process  categories.  The 
collection  instruments  used  in  this  research  seek  to  estimate  education  application  by 
collecting  seven  different  types  of  data  relating  to  the  1ST  education  and  subsequent  SEP 
billet  responsibilities.  These  collection  types  are  presented  in  Table  7. 
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Collection 

Objective 

Education 

Requirements 

Determine  the  knowledge  and  skills  required  to  perform  9648  MOS 

SEP  billet  responsibilities. 

Education 

Usefulness 

Assess  the  relative  importance  of  various  1ST  education  topics  to  9648 
MOS  SEP  billet  responsibilities. 

Education  Interest 

Assess  the  relative  subject  interest  in  various  1ST  education  topics. 

Education  Source 

Assess  the  relative  value  of  knowledge  and  skills  to  9648  MOS  SEP 
billet  responsibilities  based  on  formal  education,  self-education,  and 
social  contact. 

Education  Usage 

Assess  the  relative  frequency  and  significance  of  using  various  1ST 
educational  components  to  perform  9648  MOS  SEP  billet 
responsibilities. 

Education  in 

Support  of  Success 

Estimates  the  demonstrated  skills  and  knowledge  that  help  achieve  a 

9648  MOS  SEP  billet  success. 

Impediments  to 
Education 

Application 

Estimation  of  impediments  to  the  application  of  1ST  education  in 
information  technology  initiatives. 

Table  7.  Collection  Objectives 


C.  COLLECTION  INSTRUMENT 

A  survey  was  designed  to  gather  data  on  the  application  of  1ST  education  in 

follow-on  billets.  This  survey  was  disseminated  by  e-mail  to  all  9648  MOS  officers  for 
which  an  address  was  available.  This  survey  consists  of  a  series  of  web  pages  that  are 
made  up  of  checkboxes,  rating  buttons,  and  short  answers  fields  that  populate  a  database. 
The  survey  response  period  was  twenty-three  days — 19  September  2002  through  11 
October  2002.  The  remainder  of  this  section  describes  the  format  and  objective  of  each 
survey  page. 

1.  Respondent  Information 

The  first  page  of  the  survey  collects  information  about  the  respondent.  In  general, 
this  infonnation  is  used  to  sub-divide  the  respondent  population  based  on  the  categories 
listed  in  Table  5.  This  survey  page  has  four  sections  that  are  presented  in  Table  8.  Data 
from  this  section  are  used  to  analyze  response  data  based  on  various  subject  attributes. 
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Section  Rationale 


Respondent  information 

•  Name 

•  E-mail  address 

•  NPS  graduation  date 

Record  respondent  information  for 
further  data  collection.  Graduation 
data  are  used  to  provide  an  option 
to  divide  population  by  graduation 
date. 

What  best  describes  your  SEP  experience? 

•  Currently  in  first  SEP  billet 

•  Currently  in  second  SEP  billet 

•  Currently  in  a  post-SEP  billet 

•  SEP  payback  deferred  until  after  current  billet 

•  Never  served  in  a  SEP  billet 

Data  used  to  sub-divide  subject 
population. 

Description  of  most  recent  SEP  billet 

•  Billet  name 

•  Command 

•  Start  date 

•  End  date 

Billet  data  used  to  associate  survey 
data  with  specific  SEP  billets. 

To  what  extent  did  NPS  provide  the  tools  required  in 
your  SEP  billet? 

•  Seven  rating  buttons  with  a  range  from  Fully 
prepared  (7)  to  Unprepared  (1) 

Data  used  to  assess  the  overall 
value  of  NPS  education  to  follow- 
on  SEP  billet  requirements. 

Table  8.  Survey  Respondent  Information 


2.  Naval  Postgraduate  School  Education 

The  second  page  of  the  survey  collects  data  concerning  the  respondent’s 
education  at  NPS.  It  contains  a  series  of  checkboxes  used  to  specify  if  and  how  the 
respondents  specialized  their  NPS  education  beyond  the  default  1ST  curriculum. 
Selections  include  Decision  Support,  Modeling  and  Simulation,  Networks,  Security, 
Software  Engineering,  and  Other.  In  general,  these  data  provide  a  means  to  sub-divide  the 
subject  population  based  on  their  specific  educational  background.  Specifically,  the 
survey  outcomes  are  used  to  analyze  a  hypothesis  that  respondents  place  a  prejudicial 
value  on  topics  that  they  invested  additional  study. 

3.  Naval  Postgraduate  School  Thesis 

The  third  page  of  the  survey  collects  data  pertaining  to  the  respondent’s  NPS 

thesis.  The  page  consists  of  a  series  of  text  fields  to  gather  information  about  the  value  of 

respondents’  theses  to  the  Marine  Corps.  Table  9  describes  the  survey  page’s  three 

sections.  While  NPS  theses  are  tied  to  the  1ST  ESR  as  an  educational  objective  relating  to 
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research  skills,  these  data  are  used  to  analyze  the  extent  that  1ST  theses  provide  direct 
value  to  the  Marine  Corps  or  other  organizations. 


Section 

Rationale 

What  was  your  NPS  thesis  topic? 

Records  the  respondent’s  thesis 
topic  description. 

Did  the  Marine  Corps  benefit  from  your  NPS  thesis  by 

Short  answer  data  are  used  to 

using  it  in  some  way?  Yes. 

assess  how  NPS  thesis  research 

•  Where  and  how  was  it  used? 

positively  affects  the  Marine 

Corps.  These  data  serve  two 

Estimate  the  benefits  to  the  Marine  Corps  or  other 

purposes. 

organization 

•  Provide  for  valuation  of 

•  Cost  savings  (dollars) 

NPS  thesis  independent  of 

•  Increase  in  efficiency  (time  saving) 

subsequent  SEP  billet. 

•  Increase  in  productivity  (increased  output) 

•  Correlate  thesis  impact  and 
subsequent  SEP  billet 

•  Improvement  in  quality 

impact. 

•  Other  measure  of  benefit 

Did  the  Marine  Corps  benefit  from  your  NPS  thesis  by 

Short  answer  data  used  to  reveal 

using  it  in  some  way?  No. 

why  an  NPS  thesis  does  not  add 

Why  was  it  not  used? 

value  to  the  Marine  Coips. 

•  Did  not  generate  interest 

•  Not  a  Marine  Corps  topic 

•  Results  not  consistent  with  Marine  Corps 

objectives 

•  Other 

Table  9.  Survey  Collection  of  Thesis  Data 


4.  Education  Usefulness 

The  fourth  page  of  the  survey  collects  data  on  the  usefulness  of  1ST  education  in 
post-education  billets.  Data  consist  of  respondents’  ratings  across  the  fourteen  1ST  topics 
as  they  pertain  to  SEP  and  non-SEP  billets.  The  fourteen  topics  were  described  earlier  in 
this  chapter  and  relate  specifically  to  the  NPS  1ST  curriculum.  Table  10  describes  the 
collection  instrument.  These  data  are  used  to  assess  the  relative  value  between  education 
topics. 
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Topics 


Rating 


Rationale 


Acquisition/  Program 
Management 

Command  and  Control 

Communication  System 
Engineering 

Computer  Networking 

Computer/  Network  Security 

Database 

Decision  Support  Systems 

Economic/  Financial 
Management 

Managing  Change 

Professional  Military 
Education 

Software  Engineering 

Software  Language 
Programming 

Space  Operations 

Statistical  Analysis 


For  each  topic,  1ST  program 
topics  are  rated  in  terms  of 
their  usefulness  in  SEP  and 
non-SEP  billets. 

For  each  type  of  billet,  seven 
rating  buttons  are  provided 
with  a  range  from  Critical  (7) 
to  Not  critical  (1). 

A  non-applicable  option  is 
available  if  the  respondent 
never  studied  the  topic  or  if 
they  never  served  in  that  type 
of  billet. 

For  each  topic,  a  checkbox  is 
provided  to  indicate  if  the 
respondent’s  knowledge  of 
the  topic  was  not  provided  by 


Rating  data  are  used  to 
determine  the  usefulness  of 
1ST  program  topics  to  post¬ 
graduation  assignments. 

The  usefulness  ratings  are 
divided  between  SEP  and 
non-SEP  billets  to  assess 
variations  in  how  education  is 
used  in  different  billets. 

Data  concerning  whether  the 
topic  was  studied  at  NPS  is 
used  to  explicitly  tie  the 
education  to  the  NPS  ITM 
education. 


NPS. 


Table  10.  Survey  Collection  of  Education  Usefulness 


5.  Interest  in  Topic 

The  fifth  page  of  the  survey  collects  data  concerning  the  respondents’  interest  in 
particular  1ST  curriculum  topics.  The  page  contains  the  same  fourteen  1ST  curriculum 
topics  listed  in  Table  10.  For  each  topic,  the  respondents  rate  their  interest  in  the  topic  on 
a  seven-point  scale  ranging  from  high  (7)  to  low  (1).  Data  from  this  page  are  used  to 
examine  potential  relationships  between  respondents’  ratings  of  topic  usefulness  and 
respondents’  interest  in  topic. 


6.  Educational  Value 

The  sixth  page  of  the  survey  collects  data  on  the  value  of  the  graduate  education 
experience.  The  survey  queries  respondents  to  assess  the  relative  value  of  practical  skills, 
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theoretical  knowledge,  and  social  relationships  developed  during  graduate  education  as 
they  apply  to  SEP  billet  responsibilities.  Seven  rating  buttons  with  a  range  from  Critical 
(7)  to  Not  critical  (1)  are  provided  in  addition  to  a  Non-applicable  option.  These  three 
NPS  artifacts  (practical  skills,  theoretical  knowledge,  and  social  relationships)  are  further 
divided  to  determine  the  extent  that  the  skills,  knowledge,  and  relationships  developed 
because  of  the  NPS  experience  or  due  to  the  respondent’s  self-education.  Table  11 
describes  the  three  sections  of  the  survey  page.  These  data  are  used  to  evaluate  two 
questions. 


Section 

Rationale 

Assess  the  value  of 
practical,  hands-on 
information  technology 
skills  to  your  SEP  billet 
responsibilities. 

These  data  are  used  to  estimate  the  extent  that  practical 
information  technology  skills  are  important  to  9648  MOS  SEP 
billet  responsibilities.  The  list  of  most  valuable  skills  provides 
specific  detail. 

The  distinction  between  practical  skill  development  at  NPS  and 
practical  skills  acquired  on  own  is  used  to  explore  the  settings 
where  practical  information  technology  skills  are  developed. 

Assess  the  value  of 
theoretical  or  conceptual 
Information  Technology 
knowledge  to  your  SEP 
billet  responsibilities. 

These  data  are  used  to  estimate  the  extent  that  theoretical  and/or 
conceptual  information  technology  knowledge  is  important  to 

9648  MOS  SEP  billet  responsibilities.  The  list  of  most  valuable 
theoretical  knowledge  provides  specific  detail. 

The  distinction  between  theory  learned  at  NPS  and  theory  learned 
on  own  is  used  to  explore  the  settings  where  theoretical 
knowledge  of  information  technology  is  developed. 

Assess  the  value  of  your 
network  of  professional 
contacts  that  resulted 
from  your  NPS 
experience  to  your  SEP 
billet  responsibilities. 

These  data  are  used  to  estimate  the  extent  that  social  relationships 
developing  from  resident  NPS  education  have  on  9648  MOS  SEP 
billet  responsibilities. 

The  distinction  between  fellow  students  and  NPS  personnel  is 
used  to  differentiate  the  types  of  social  interactions  that  are  useful. 

Table  1 1 .  Survey  Collection  of  Education  Value 


7.  Education  Usage 

The  seventh  page  of  the  survey  builds  upon  the  previous  page  and  collects  data  on 
the  significance  and  frequency  of  employing  different  education  components  listed  in 
Table  12.  It  queries,  “During  your  SEP  toru,  how  often  did  you  call  on  the  following,  and 
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how  important  was  it?”  Table  12  lists  the  educational  components,  rating  options,  and 
rationale. 


Component 

Rating 

Rationale 

Skills  acquired  at  NPS 

Knowledge  developed  at  NPS 

1ST  course  material 

NPS  resources  like 
professors,  staff,  or  current 
students 

Fellow  NPS  students 

For  each  component,  rate  the 
frequency  that  it  is  used. 

Rating  options  include  daily, 
weekly,  monthly,  quarterly, 
yearly,  and  never. 

For  each  component,  rate  the 
value  of  the  component  to 
your  9648  MOS  SEP  billet 
responsibilities.  Seven  rating 
buttons  provide  a  range  from 
Critical  (7)  to  Not  critical  (1). 

These  data  are  used  to 
compare  education  value  and 
usage  frequency.  It  provides 
for  the  collection  of  low 
value  education  components 
that  may  have  high  usage 
rates. 

Table  12. 

Survey  Collection  of  Education  Usage 

8.  Billet  Success 

The  eighth  page  of  the  survey  collects  data  relating  to  successful  SEP  initiatives 
that  can  be  traced,  in  part,  to  the  respondents’  NPS  education.  Data  are  used  to  identify 
core  SEP  billet  processes  and  specific  educational  components  that  demonstrated  value. 
The  survey  collects  a  description  of  the  success,  a  description  of  the  impact  of  the  success 
on  the  Marine  Corps,  and  an  assessment  of  how  NPS  education  aided  in  the  success.  The 
assessment  provides  for  a  description  of  employed  skills,  knowledge,  and  other  education 
that  helped  achieve  the  success.  These  data  are  used  to  correlate  the  value  generated  in 
specific  SEP  billet  actions  to  specific  elements  of  NPS  education. 

9.  Information  Technology  Failures 

The  ninth  page  of  the  survey  balances  the  previous  page  by  collecting  data  on 
failed  infonnation  technology  (IT)  initiatives.  Failures  are  limited  to  the  respondents’ 
9648  MOS  SEP  billet  experience.  The  survey  collects  a  description  of  the  failed  IT 
initiative  and  the  respondents’  assessment  of  the  reasons  that  the  initiative  failed.  The 
reasons  include  failure  to  demonstrate  value,  funding,  insufficient  time,  lack  of 
organizational  buy-in,  technical  complexity,  training,  and  other.  These  data  are  used  to 
analyze  what  9648  MOS  SEP  billet  factors  prevent  greater  positive  impact  of  graduate 
1ST  education. 
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10.  Comments 

The  tenth  and  final  page  of  the  survey  asks  respondents  if  they  are  willing  to 
conduct  a  follow-up  telephone  interview  and  provides  a  text  field  for  additional 
comments. 
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Ill  DATA  ANALYSIS 


This  chapter  analyzes  respondent  survey  data  to  answer  the  research  questions. 

A.  LIMITATIONS 

The  survey  achieved  the  collection  objectives  described  in  Chapter  II.  However, 
there  are  some  inherent  limitations  to  analysis  based  on  the  available  data.  Most  of  the 
limitations  can  be  attributed  to  the  primary  data  sources:  Billet  Education  Evaluation 
Certificates  (BEEC)  and  survey  responses  from  Marine  officers  having  the  9648  Military 
Occupation  Specialty  (MOS). 

1.  Billet  Education  Evaluation  Certificates  Data 

The  Marine  Corps  Total  Force  Structure  Process  mandates  that  all  Special 
Education  Program  (SEP)  billets  have  a  validated  BEEC.  As  explained  in  the  previous 
chapter,  a  BEEC  defines  the  graduate  level  education  requirements  for  a  specific  billet  or 
job.  Fifty-five  9648  MOS  billets  require  a  BEEC;  however,  only  twenty-four  usable 
BEECs  were  collected. 

While  over  fifty  9648  MOS  BEECs  were  collected  from  the  Program  Sponsor  and 
the  Total  Force  Structure  Division,  more  than  half  were  dismissed  due  to  age  and 
duplication.  The  age  cutoff  used  in  this  screening  is  five  years,  despite  the  requirement 
for  bi-annual  recertification. 

Additionally,  the  specificity  of  the  education  requirements  listed  on  the  BEECs 
varies  greatly.  While  some  appeared  thoughtfully  developed,  several  simply  stated  that 
billet  education  requirements  were  equal  to  the  Naval  Postgraduate  School  (NPS) 
Infonnation  System  Technology  (1ST)  curriculum.  In  fact,  one  even  called  for  the  NPS 
Computer  Science  curriculum  (which  provides  the  graduate  with  a  9646  MOS).  For  these 
reasons,  most  of  the  collected  BEECs  fall  short  of  complying  with  the  Marine  Corps’ 
graduate  education  certification  intent,  and  they  do  not  meet  data  collection  expectations. 

The  Program  Sponsor  stated  that  the  BEEC  validation  process  is  very  low  priority 

and,  in  practice,  does  not  inhibit  SEP  functions  (Klinger  2002a).  It  is  difficult  to 

determine  if  the  BEEC  process  is  useful  because  it  is  simply  not  managed  and  enforced 

consistent  with  established  policies.  Data  from  the  collected  BEECs  are  considered 
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during  analysis,  but  they  are  not  valued  strongly  because  they  are  incomplete  and  of 
inconsistent  substance. 

2.  Survey  Data 

The  web-based  survey  was  disseminated  to  eighty-four  marine  officers  who  hold 
the  9648  MOS,  and  it  collected  responses  for  twenty-three  days.  The  survey  returned  data 
from  forty-four  subjects — fifty-two  percent.  However,  not  every  respondent  completed 
the  entire  survey.  Thirty-four  and  half  percent  or  twenty-nine  of  the  eighty-four  subjects 
completed  the  entire  survey.  This  response  rate  actually  surpassed  expectations;  however, 
the  responses  are  subjective  and  thus  limit  responsible  analysis. 

The  survey  responses  contained  an  unexpected  element  that  reduces  the 
conclusiveness  of  some  analysis.  The  survey  responses  proved  very  difficult  to  pair  with 
available  BEECs.  Of  the  twenty-four  BEECs  used  in  this  study,  only  half  can  be  paired 
with  survey  respondents.  These  twelve  BEECs  associate  with  seventeen  survey  responses 
because  several  respondents  held  common  billets.  But  most  responses  cannot  be 
associated  to  a  BEEC,  so  it  is  difficult  to  conclusively  assess  how  well  BEECs  describe 
the  education  requirements  for  specific  billets. 

B.  CURRICULUM  FULFILLMENT  OF  EDUCATION  REQUIREMENTS 

The  1ST  curriculum  is  tailored  to  provide  graduate  level  Information  Technology 

(IT)  education  to  military  officers.  The  Marine  Corps  articulates  its  graduate  level  IT 
educational  requirements  by  specific  billet.  These  education  requirements  are  described 
using  a  BEEC  for  every  billet  requiring  the  9648  MOS.  It  is  reasonable  to  assume  that  the 
educational  topics  contained  in  the  BEECs  should  resemble  the  course  topics  that  make 
up  the  1ST  curriculum.  Furthermore,  survey  data  on  the  value  of  education  topics  in  9648 
billets  should  be  consistent  with  both  the  BEECs  and  the  1ST  curriculum.  A  valuation  of 
the  1ST  curriculum  can  be  assessed,  in  part,  by  how  well  it  aligns  with  BEEC 
requirements  and  by  testaments  of  officers  who  used  the  1ST  education  in  9648  MOS 
billets. 
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1.  Formal  Education  Requirements  and  Curriculum  Composition 

The  fourteen  education  topics  that  make-up  the  1ST  curriculum  meet  nearly  all  of 

the  education  topic  requirements  listed  in  the  twenty-four  9648  MOS  BEECs  that  were 
collected  in  this  study.  Only  one  topic  was  listed  on  BEECs  that  are  not  part  of  the  default 
1ST  curriculum  courses:  web  application  technologies.  However,  the  Department  of 
Information  Sciences  does  provide  elective  courses  on  this  topic.  Notwithstanding  this 
topic,  these  data  support  the  assertion  that  the  1ST  curriculum  provides  the  education 
topics  required  by  the  Marine  Corps.  Figure  8  plots  the  frequency  (by  percent)  that  each 
of  the  1ST  education  topics  are  identified  on  the  twenty-four  9648  MOS  BEECs. 


Rate  of  Topic  Occurrance  on  BEECs 
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UJ  Decision  Support  Systems 

Softw  are  Language  Programming 
Command  and  Control 
Space  Operations 
Professional  Military  Education 
Web  Technologies  * 


*  Not  part  of  the  default  1ST  curriculum. 


Percent  Occurrance  (n=24) 


Figure  8.  Rate  of  Topic  Occurrence  on  Education  Certificates 


While  Figure  8  indicates  that  the  1ST  curriculum  includes  most  of  the  education 
topics  required  by  the  Marine  Corps,  it  does  not  reveal  how  well  the  curriculum  meets 
these  requirements.  If  the  1ST  curriculum  is  tuned  to  meet  the  education  requirements 
contained  in  the  BEECs,  it  is  reasonable  to  expect  that  the  curriculum  time  devoted  to 
each  education  topics  should  be  roughly  proportional  to  the  BEEC  occurrence  frequency. 
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Figure  9  depicts  this  relationship  and  reveals  that  the  1ST  curriculum  is  not 
uniformly  proportioned  to  meet  the  aggregate  BEEC  education  requirements.  Figure  9 
plots  the  difference  between  education  topic  occurrences  from  the  BEECs  and  the 
number  of  1ST  course  hours  devoted  to  each  topic  (topic  occurrence  frequency  minus 
total  topic  course  hours).  It  highlights  three  conditions  where  BEEC  education 
requirements  are  under  represented,  balanced,  and  over  represented  by  the  1ST 
curriculum.  There  is  no  explicit  value  used  to  differentiate  between  the  balanced  and 
unbalanced  conditions,  but  rather  the  figure  serves  to  highlight  the  relativity  of  the 
differences. 


Education  Topic  Comparison  of  BEEC 
Occurrence  and  Course  Hours 


Figure  9.  Comparison  of  Education  Requirements  and  Curriculum  Course  Hours 

It  is  important  to  note  that  the  data  supporting  this  graph  is  not  normalized  and  is 
not  expected  to  show  centrality  about  zero.  Additionally,  this  comparison  assumes  that 
meeting  education  topic  requirements  can  be  achieved  in  a  common  number  of  course 
hours,  regardless  of  topic.  However,  it  is  sensible  to  assume  that  high-frequency 
requirements  should  receive  higher  curriculum  weighting  and  suggests  that  differences 
larger  than  about  five  indicate  less  than  appropriate  alignment. 
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Figure  9  shows  that  most  of  the  1ST  topics  are  balanced.  It  also  reveals  that 
Communication  System  Engineering  receives  significantly  more  emphasis  in  the  1ST 
curriculum  than  expected  from  the  stated  requirements.  Likewise,  two  topics  do  not 
appear  to  receive  the  course  hour  weighting  appropriate  to  the  BEEC  requirements. 

2.  Graduate  Assessments  and  Curriculum  Composition 

Thirty-eight  survey  respondents  rated  the  usefulness  of  the  fourteen  1ST  education 

topics.  Figure  10  plots  the  average  respondent  rating  by  topic;  there  are  no  significant 
outliers  in  the  response  data.  This  figure  reveals  that  the  topics  of  the  1ST  curriculum 
meet  the  Marine  Corps’  9648  MOS  education  requirements,  this  time  described  by 
survey  respondents. 


Average  Assessment  of  Topic  Usefulness 


Average  Survey  Response 

(n=38,  scale:  0-7) 
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Managing  Change 
Professional  Military  Education 
Software  Engineering 
Software  Language  Programming 
Space  Operations 
Statistical  Analysis 


Average:  4.4 


Figure  10.  Respondent  Assessment  of  Topic  Value 

Figure  10  indicates  that  Acquisition  and  Program  Management  is  the  most  useful 

education  topic  in  9648  MOS  billets;  conversely,  Space  Operations  is  the  least  useful. 

Similar  to  the  comparison  in  Figure  9,  Figure  1 1  plots  the  comparison  of  these  average 

ratings  against  the  number  of  course  hours  in  each  1ST  topic.  In  this  graph,  course  hours 
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have  been  divided  in  half  to  help  normalize  the  data  for  comparison.  It  highlights  three 
conditions  where  BEEC  education  requirements  are  under  represented,  balanced,  and 
over  represented  by  the  1ST  curriculum. 


Comparison  of  Topic  Usefulness  to  Course  Load 


Average  Usefulness  Rating  minus  Half  Course  Hours 

Figure  11.  Comparison  of  Average  Topic  Value  and  Curriculum  Course  Flours 

Upon  reviewing  the  plots  in  Figure  9  and  Figure  11,  several  similarities  become 
apparent.  Using  two  different  methods  to  identify  education  requirements  (BEECs  and 
survey  responses),  the  1ST  curriculum  course  hours  appear  proportional  to  about  half  of 
the  education  requirements.  Most  striking  is  the  education  topic  that  is  significantly  over 
represented  in  both  comparisons:  Communication  System  Engineering.  Interestingly,  the 
Department  of  Information  Sciences  is  currently  redesigning  the  Communication  System 
Engineering  education  topic,  and  the  course  hours  devoted  to  this  topic  will  be  cut  by 
nearly  fifty  percent.  The  decision  to  redesign  this  topic  resulted  from  a  student-led  effort 
that  demonstrated  low  usage  of  communication  system  engineering  by  curriculum 
graduates. 

While  somewhat  less  conclusive,  these  figures  also  suggest  topics  that  may  be 
under  represented  in  the  1ST  curriculum.  In  both  comparisons,  Acquisition  and  Program 
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Management,  Computer  Networking,  and  Database  rank  as  the  top  three  topics  where 
requirements  exceed  the  course  hour  allocation. 

It  should  be  remembered  that  these  comparisons  are  not  expected  to  align 
completely.  Furthermore,  both  the  BEECs  and  survey  responses  have  an  historical  bias. 
In  the  case  of  the  BEECs,  the  certification  of  the  requirements  takes  several  years. 
Similarly,  but  to  a  lesser  extent,  survey  respondents  assessed  the  education  topics  based 
on  their  9648  MOS  billet  experience,  which  tends  to  be  a  few  years  old.  Moreover,  the 
1ST  curriculum  is  current  and  evolving  to  provide  education  that  will  be  used  in  the 
future.  Therefore,  it  must  be  acknowledged  that  the  Department  of  Information  Sciences 
tries  to  anticipate  emerging  education  topic  requirements  and  factors  these  projections 
into  curriculum  design,  which  could  explain  some  of  the  comparison  disparities. 

C.  INFORMATION  SYSTEM  TECHNOLOGY  EDUCATION  EFFECTS 

The  Marine  Corps’  continued  investment  in  the  NPS  1ST  curriculum  suggests  that 

the  Corps  perceives  a  positive  organizational  impact  stemming  from  this  educational 
investment.  In  fact,  curriculum  enrolment  has  increased  over  the  past  several  years. 
Survey  data  allows  for  the  effects  of  1ST  education  to  be  analyzed  in  the  three  domains 
where  it  affects  the  Marine  Corps:  pre-graduation,  SEP  billets,  and  non-SEP  billets.  The 
survey  data  suggest  that  the  1ST  curriculum  can  and  often  does  positively  affect  the 
Marine  Corps  in  each  of  these  domains. 

1.  Pre-graduation  Effects  of  Education 

Officers  in  the  1ST  curriculum  can  affect  the  Marine  Corps  through  their  thesis 
research.  However,  thesis  research  provides  little  value  to  the  Marine  Corps  when  the 
research  is  not  sponsored.  Twenty-one  respondents  indicate  that  their  NPS  1ST  thesis 
provided  no  value  to  the  Corps.  Two  elements  are  common  in  eighteen  of  the  twenty-one 
responses:  the  Maine  Corps  did  not  sponsor  the  thesis  and/or  the  thesis  topic  was  not 
related  specifically  to  the  Marine  Corps.  Seven  other  respondents  indicate  that  they  do 
not  know  if  their  thesis  work  provides  value  to  the  Corps.  It  is  reasonable  to  suspect  that 
their  efforts  were  of  no  significant  benefit  to  the  Marine  Corps. 

However,  there  are  fourteen  cases  when  respondents  felt  that  their  NPS  thesis 
research  did  provide  value  to  the  Marine  Corps.  The  trend  in  each  of  these  cases  is  that 
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the  thesis  had  a  Marine  Corps  or  other  Marine  Corps  supporting  activity  sponsoring  the 
research.  Of  the  respondents  that  believe  their  thesis  provided  value,  the  type  of  value 
broke  down  as  follows. 

•  One  thesis  produced  a  scheduling  tool  that  has  been  deployed  to  Marine 
Corps  recruiters.  The  respondent  estimated  that  the  tool  saved  every 
recruiter  sixty  hours  a  year.  Conservatively,  this  translates  to  between  four 
and  five  hundred  dollar  of  salary  cost  saving  per  recruiter  per  year. 
Perhaps  more  significant  than  the  salary  cost,  the  time  saved  provides  the 
recruiter  more  time  to  focus  on  his  mission  and  locate  higher  quality 
recruits  for  the  service. 

•  Three  theses  provided  value  buy  aligning  thesis  study  with  follow-on  billet 
assignment.  In  these  cases,  the  value  resulted  from  increased  efficiency 
when  the  officer  arrived  at  the  billet.  While  it  is  difficult  to  assign  a  value 
to  this,  it  clearly  speeds  the  time  required  to  transition  into  a  new  position, 
which  suggests  greater  overall  productivity. 

•  Three  theses  provided  value  by  supporting  concurrent  Marine  Corps 
activities  involving  planning  and  studies.  In  these  cases,  the  Marine  Corps 
avoided  costs  by  using  thesis  work  instead  of  external  consulting. 

•  Four  theses  provided  indirect  value  to  the  Marine  Corps  by  advancing 
research  efforts  by  activities  that  support  the  Marine  Corps.  The  value  of 
these  efforts  are  found  in  the  Marine  Corps  ability  to  influence  advanced 
research  through  the  use  of  Marine  officers  who  retain  a  vested  interest  in 
the  research  and  who  retain  the  research  knowledge  for  subsequent 
assignments. 

These  analyses  suggest  that  the  Marine  Corps  could  better  leverage  the  potential 
to  attain  greater  value  from  NPS  theses  by  providing  wider  sponsorship.  Currently,  there 
is  no  Marine  Corps  activity  that  channels  research  requirements  to  Marine  students  at  the 
NPS.  Most  thesis  sponsorship  develops  from  student-initiated  contact  with  potential 
sponsors. 

2.  Curriculum  Effects  on  Designated  Information  Technology  Billets 

Analysis  indicates  that  the  1ST  curriculum  contributes  positively  to  graduates’ 

performance  in  9648  MOS  billets.  This  should  not  be  surprising  because  these  billets 
have  been  certified  as  requiring  specific  elements  of  the  1ST  curriculum.  While  survey 
data  cannot  show  specific  effects,  it  is  reasonable  to  conclude  that  survey  responses  that 
ascribe  high  value  to  elements  of  the  1ST  curriculum,  as  they  affect  9648  MOS  billet 
responsibilities,  should  favorably  affect  the  Marine  Corps.  The  values  of  several  different 

elements  of  the  1ST  curriculum  are  presented  in  this  section. 
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Seventy-three  percent  of  survey  respondents  reported  that  the  1ST  curriculum 
adequately  prepared  them  for  their  subsequent  9648  MOS  responsibilities.  The  response 
options  for  this  query  consisted  of  a  seven-point  rating  scale  with  seven  representing 
“fully  prepared”  and  one  representing  “unprepared.”  The  term  adequately  is  used  to 
describe  response  ratings  of  five,  six,  and  seven.  The  distribution  of  these  ratings  is 
plotted  in  Figure  12. 


Survey  respondents  were  also  asked  to  evaluate  each  1ST  curriculum  topic  with 
respect  to  their  9648  MOS  responsibilities.  These  data  were  presented  in  Figure  10  by  the 
value  ascribed  to  each  education  topic.  Figure  13  plots  the  distribution  of  the  same 
response  data  in  an  aggregated  form. 

Figure  13  reveals  that  fifty-seven  percent  of  the  1ST  curriculum  topics  were  useful 
to  respondents’  9648  MOS  responsibilities.  Fifty-seven  percent  seems  to  suggest  that  the 
education  topics  are  not  particularly  useful.  However,  these  ratings  account  for  all 
fourteen  education  topics,  many  of  which  were  not  highly  valued.  To  help  place  this 
analysis  in  context,  Acquisition  and  Program  Management  was  the  most  highly  valued 
education  topic;  eighty-four  percent  of  respondents  reported  high  value  for  this  topic.  On 
the  other  extreme,  Space  Operations  received  the  lowest  value  rating;  only  sixteen 
percent  of  respondents  reported  high  value.  Therefore,  the  fifty-seven  percent  rating 
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across  all  topics  indicates  that  most  of  the  1ST  education  topics  are  useful  in  9648  MOS 
billets. 


Figure  13.  Respondent  Assessment  of  Education  Usefulness  by  Topic 

It  is  not  surprising  to  see  that  individual  1ST  curriculum  topics  are  not  consistently 
useful  across  all  9648  MOS  billets.  The  1ST  curriculum  is  designed  to  meet  a  broad  set  of 
learning  objectives.  Central  to  the  curriculum  is  the  goal  to  provide  students  with 
theoretical  knowledge  relating  to  information  system  technology.  Figure  14  plots  the 
distribution  of  survey  respondent  assessments  of  how  well  the  1ST  curriculum  achieves 
this  goal.  The  response  data  overwhelmingly  suggests — eighty-five  percent — that  the  1ST 
education  provides  knowledge  that  is  valuable  to  subsequent  9648  MOS  responsibilities. 
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A  secondary  objective  of  the  1ST  curriculum  is  the  development  of  practical 
information  technology  skills.  While  subordinate  to  developing  theoretical  knowledge, 
these  skills  also  provide  value.  Figure  15  plots  the  distribution  of  survey  respondent 
assessments  of  how  well  the  1ST  curriculum  achieves  this  goal.  With  nearly  the  same 
enthusiasm  as  the  previous  figure — seventy-four  percent — respondents  report  value  to 
their  9648  MOS  responsibilities  from  practical  skills  acquired  at  NPS. 


Figure  15.  Value  of  Practical  Skills 
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A  final  indication  of  the  value  of  the  1ST  education  to  the  Marine  Corps  can  be 
established  by  determining  how  often  the  education  is  actually  used.  Figure  16  plots  the 
distribution  of  respondents’  assessment  of  education  usage  frequency.  It  indicates  that 
more  than  sixty  percent  of  survey  respondents  feel  that  they  use  the  knowledge  and  skills 
acquired  from  the  1ST  curriculum  on  a  daily  basis.  This  high  rate  of  usage  coupled  with 
the  high  value  assigned  to  the  1ST  education  (Figure  14  and  Figure  15)  demonstrates  the 
clearest  example,  from  the  available  data,  that  the  NPS  1ST  education  significantly 
impacts  the  Marine  Corps  9648  MOS  billets. 


Frequency  of  Education  Usage 


Recurrence  of  Usage 

Figure  1 6.  Respondent  Assessment  of  Education  Usage 

A  final  way  that  respondents  indicated  how  the  1ST  education  positively  affected 
the  Marine  Corp  is  through  descriptions  of  their  successful  information  technology 
initiatives.  Thirty  respondents  indicted  that  they  were  responsible  or  contributed  to 
successful  information  technology  initiatives  as  part  of  their  9648  MOS  billet.  The  types 
of  successes  range  from  improved  service,  to  better  acquisition,  to  effective  transition. 
While  the  survey  did  not  capture  the  monetary  value  of  these  successes,  the  complexity 
associated  with  modern  military  information  system  technology  suggests  that  the  1ST 
education  contributed  to  the  eventual  success  of  each  of  these  cases. 

The  analysis  described  in  this  section  seems  to  validate  the  Marine  Corps 
management  of  its  graduate  level  information  technology  education.  Despite  the  fact  that 

there  is  not  a  responsive  feedback  mechanism  in  place  to  determine  how  well  these 
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education  investments  affect  the  organization,  these  analyses  suggest  that  the  graduate 
education  requirements  and  provisioning  mechanisms  achieve  most  of  the  service 
objectives. 

3.  Effects  of  Education  on  General  Billets 

The  benefits  of  the  1ST  education  are  not  solely  tied  to  9648  MOS  billets.  Many 
officers  with  the  9648  MOS  return  to  their  primary  MOS.  This  section  reveals  how  the 
1ST  education  is  valued  in  these  non-9648  MOS  billets.  While  the  previous  section 
demonstrated  that  the  1ST  education  provides  substantial  value  to  9648  MOS  billets,  this 
section  shows  how  this  value  does  not  significantly  fade  in  other  MOSs. 

Figure  17  plots  the  distribution  of  respondents’  assessment  of  education  topic 
value  to  non-9648  MOS  billet  responsibilities.  This  figure  is  similar  to  Figure  13,  except 
it  references  non-9648  MOS  billets.  Perhaps  not  surprisingly,  it  indicates  that  1ST 
education  topics  are  generally  not  as  useful  in  non-9648  MOS  billets.  It  shows  that  only 
forty-six  percent  of  the  respondents  felt  that  1ST  topics  were  valuable  in  these  billets 
compared  to  the  fifty-seven  percent  who  felt  that  the  topics  were  valuable  in  9648  billets. 


While  there  is  generally  a  lower  value  placed  on  topic  value  in  non-9648  MOS 

billets,  there  is  a  significant  similarity  between  the  ratings.  When  comparing  the  value  of 

education  topics  to  9648  and  non-9648  billets  the  figures  indicate  that  sixty-seven  percent 

of  the  280  ratings  agree  to  within  one  rating  point. 
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The  most  interesting  piece  of  data  from  this  comparison  is  the  identification  of 
1ST  education  topics  that  were  ascribed  with  equal  or  greater  value  in  non-9648  MOS 
billets  than  in  the  9648  MOS  billets.  When  the  average  education  topic  ratings  for  9648 
and  non-9648  MOS  billets  are  compared,  five  topics  have  a  difference  that  shows  greater 
value  in  non-9648  MOS  billets:  Professional  Military  Education,  Command  and  Control, 
Computer  Networks,  Communications  Engineering,  and  Space  Operations.  However, 
closer  review  of  these  relationships  seems  to  indicate  that  these  rating  differences  may 
stem  from  low  9648  MOS  ratings  as  opposed  to  high  non-9648  MOS  ratings.  In  other 
words,  these  topics  were  generally  rated  low  in  both  categories,  whereas  the  other  topics 
were  rated  high  for  their  value  to  9648  MOS  billets. 

D.  EFFECT  OF  PERSONAE  PREFERENCE  ON  VALUE 

Education  value  varies  by  perspective.  The  value  of  education  within  an 

organization  may  be  different  from  the  value  placed  on  the  education  by  an  individual. 
Furthermore,  a  student’s  personal  preferences  may  influence  both  the  student’s  and  the 
organization’s  assessment  of  education  value.  This  research  has  demonstrated  some 
significant  similarities  between  the  education  topics  valued  by  the  organization — 
requirements — and  personal  testaments  on  the  value  of  those  topics.  This  section 
compares  survey  respondents’  assessments  of  education  topics  and  their  personal 
preferences  for  those  topics.  Two  methods  are  used  to  detennine  personal  preferences: 
curriculum  specialization  and  respondent  ratings  of  topic  interest.  This  analysis  shows 
that  individuals  place  higher  value  on  education  topics  that  they  have  a  personal  interest 
in  studying.  The  results  of  this  analysis  suggest  that  greater  personal  value,  and 
potentially  greater  organizational  value,  can  be  achieved  by  allowing  students  flexibility 
in  the  composition  of  curriculums. 

1.  Student  Interest  in  Education  Topics 

Thirty-seven  survey  respondents  rated  the  fourteen  1ST  education  topics  in  terms 
of  their  value  to  9648  MOS  billets  and  in  tenns  of  the  respondent’s  personal  interest  in 
the  topics.  There  was  a  moderate  correlation  between  respondents’  assessments  of  topic 
value  and  interest.  Two  graphs  are  used  to  present  these  relationships. 
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Figure  18  plots  the  mathematical  difference  between  education  topic  interest  and 
value  rating.  Based  on  the  seven-point  survey  rating  scale,  differences  can  range  from 
plus-seven  to  minus-seven.  The  figure  reveals  that  there  is  little  difference  in  respondent 
ratings  of  topic  interest  and  value.  In  fact,  sixty-five  percent  of  the  differences  are  one 
point  or  less,  and  the  average  difference  is  only  three  tenths  of  a  point.  While  this  figure 
shows  part  of  the  relationship,  it  does  not  specifically  indicate  how  preference  in  an 
education  topic  affects  perceptions  of  value. 


Comparison  of  Interest  and  Value 


Rating  Difference  (Interest  minus  Value) 


Figure  18.  Difference  in  Value  and  Student  Interest  of  Education  Topics 


Figure  19  plots  the  average  value  of  education  topics  based  on  respondent  interest 
ratings.  It  reveals  that  topic  value  is  consistently  higher  when  respondents  assess  high 
personal  interest  in  the  topic.  While  interest  alone  does  not  predict  value,  the  relationship 
suggests  that  positive  or  negative  student  interest  in  education  topics  do  have  similar 
effects  on  value  assessments.  In  this  example,  there  is  an  eighty-nine  percent  chance  that 
a  topic  will  receive  a  value  rating  of  five  or  greater,  given  an  interest  rating  of  seven. 
Conversely,  there  is  only  a  twenty-three  percent  chance  that  a  topic  will  receive  a  rating 
of  five  or  greater,  given  an  interest  rating  of  three  or  lower. 
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Value  of  Education  Topic  by  Interest  in  Topic 


Average  Value  Rating 

Figure  19.  Education  Topic  Value  by  Student  Interest  in  Topic 

2.  Student  Investment  in  Education  Topics 

The  1ST  curriculum  allows  students  a  limited  ability  to  specialize  their  studies.6 
Twenty  survey  respondents  indicated  that  they  specialized  their  1ST  education.  While  not 
collected  in  the  survey;  presumably,  these  students  felt  some  motivation  to  schedule  the 
additional  course  work.  The  specific  motivations  are  probably  as  diverse  as  the 
respondents;  however,  two  seem  reasonable:  personal  interest  and  anticipation  of  future 
need.  In  either  case,  students  expended  personal  resources — time  in  this  case — to  acquire 
additional  learning.  Table  13  summarized  the  effect  of  these  specializations  on 
assessments  of  education  topic  interest  and  perceived  topic  value. 


6  The  default  curriculum  used  in  this  study  allows  for  two  electives.  To  select  additional  courses,  students 
must  validate  default  courses,  or  over-schedule  courses.  The  1ST  curriculum  is  presently  being  redesigned 
to  incorporate  course  “tracks.”  These  tracks  will  provide  students  with  greater  opportunities  to  focus  studies 
in  specific  topic  areas. 
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Effects  of  Ed 

lucation  Specialization  on  Education  Topic  Interest  and  Value  * 

Effects  on 

Topic  Interest 

When  respondents  design  study  based  upon  particular  topics,  they  rated 
their  interest  in  those  topics  one  point  higher  than  other  topics. 

Respondents  have  a  seventy-seven  percent  chance  of  reporting  high  interest 
(ratings  five,  six,  or  seven  points)  in  topics  that  they  reported  as  specialized 
study. 

Effects  on 

Topic  Value 

When  respondents  designed  study  based  upon  particular  topics,  they  rated 
the  value  of  those  topics  one  point  higher  than  other  topics. 

Respondents  have  an  eighty-two  percent  chance  of  reporting  high  interest 
(ratings  five,  six,  or  seven  points)  in  topics  that  they  reported  as  specialized 
study. 

*  Data  from  twenty  survey  responses  that  reported  one  or  more  education  topic  specialization  beyond 
the  default  1ST  curriculum.  Ratings  based  on  a  seven-point  scale. 

Table  13.  Significance  of  Education  Specialization 


The  data  supporting  Table  13  indicates  that  decisions  to  invest  in  additional 
education  are  based,  in  part,  on  a  students’  interest  in  the  education  topic.  More 
significantly,  it  reveals  that  specialization  usually  predicts  high  valuation  of  the  education 
topic.  This  is  consistent  with  the  previous  analysis  of  interest  and  value,  where  high 
respondent  interest  assessments  usually  predicted  high  value  assessments. 


E.  SOCIAL  IMPACT  OF  RESIDENT  EDUCATION 

Many  Marine  Corps  officers  pursue  graduate  level  education.  These  education 

endeavors  range  from  member-funded  part-time  study,  to  Marine  Corps  supported  off- 
duty  education,  or  to  full-time  fully-funded  advance  degree  providing  education  like  the 
SEP.  As  mentioned  previously,  the  NPS  provides  most  of  the  Marine  Corps’  SEP 
education,  and  the  NPS  1ST  curriculum  is  the  sole  source  of  9648  MOS  qualifications. 

One  of  the  unique  characteristics  of  the  NPS  education  is  its  corporate 
environment.  The  school  is  dedicated  to  educating  military  officers.  The  school  responds 
to  the  needs  of  the  Department  of  Defense  in  tenns  of  enrolment,  curriculum,  and 
research.  This  results  in  an  academic  environment  where  students  and  staff  share  a 
common  desire  to  advance  the  national  security  posture  of  the  United  States.  Presumably, 
a  valuable  byproduct  of  graduate  education  in  this  setting  is  an  increased  social 
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connection  among  defense  professionals.  This  section  analyzes  the  extent  that  social 
relationships  developed  at  NPS  contribute  to  post-education  perfonnance. 

1.  Value  of  Naval  Postgraduate  School  Relationships 

Graduates  of  the  NPS  1ST  curriculum  indicate  that  social  relationships  developed 

at  the  NPS  are  important  to  their  subsequent  assignments.  Figure  20  plots  the  average 
respondent  rating  for  various  elements  of  their  NPS  experience.  Using  a  rating  scale  that 
ranged  from  one  (not  critical)  to  seven  (critical),  respondents  rated  how  significant  NPS 
acquired  knowledge,  skills,  peer  relationships,  and  NPS  staff  relations  were  to  their 
subsequent  billet  responsibilities. 


Value  of  NPS  Education 


Average  Rating  (0-7) 


Figure  20.  Average  Respondent  Rating  of  Significance  of  NPS  Experience 


As  shown  in  previous  sections,  the  figure  shows  that  knowledge  and  skills 
developed  at  NPS  are  quite  significant.  This  should  come  as  no  surprise,  as  the  explicit 
objective  of  NPS  is  to  provide  students  with  pertinent  knowledge  and  skills  in 
information  technology.  These  high  ratings  confirm  that  NPS  is  meeting  its  educational 
objectives.  What  may  not  be  as  intuitive  is  the  significance  that  respondents  placed  on  the 
social  relationships  advanced  during  NPS  education.  The  data  supporting  Figure  20 
indicates  that  respondents  ascribed  reasonably  high  value  to  the  formulation  of  social 
relationships  among  student  peers.  Somewhat  less  significant  is  the  value  of  relationships 
with  members  of  the  NPS  staff. 


This  may  be  explained  by  the  social  setting  at  NPS.  Students  progress  through  the 
two-year  curriculum  by  section.  Sections  are  formed  from  students  in  like  curriculums 
who  start  their  education  at  the  same  time.  Therefore,  most  students  move  through  the 
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curriculum  with  a  common  group  of  about  thirty  to  fifty  peers.  Students’  relationships 
with  NPS  staff  members  should  naturally  be  less  mature  because  most  staff  members  are 
exposed  to  students  with  a  much  lower  frequency  than  students  within  their  peer  group. 
However,  this  data  suggest  that  social  relationships  are  influential.  Furthennore,  it 
indicates  that  different  education  methods,  such  as  various  distant  learning  programs, 
may  fall  short  of  achieving  the  full  potential  of  the  resident  programs  if  these  social 
issues  are  not  considered. 

2.  Probability  of  Using  Social  Relationships 

Another  way  to  estimate  the  value  of  social  relationships  is  to  determine  the 
frequency  that  they  are  used.  Figure  21  plots  the  average  probability  that  the  thirty-two 
respondents  would  use  the  knowledge,  skills,  and  social  relationships  developed  at  NPS 
in  a  given  week.  A  comparison  between  Figure  20  and  Figure  21  reveals  that  the 
probability  of  use  is  aligned  with  the  value  of  the  education  component.  In  fact,  the 
correlation  between  value  and  probability  of  use  is  0.99. 
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Figure  2 1 .  Weekly  Probability  of  Respondent  Use  of  Education 


The  probability  of  calling  on  social  contacts  developed  at  NPS,  especially 
student-peer  relationships,  is  a  further  indication  that  social  relationships  emerging  from 
NPS  education  provide  value  to  the  Marine  Corps.  It  helps  to  quantify  the  value  of  social 
relationships  by  demonstrating  the  frequency  that  these  relationships  are  employed  in 
post-education  billets.  From  Figure  21,  there  is  about  a  sixty  percent  chance  that  an  1ST 
graduate  will  contact  a  fellow  1ST  student  in  a  given  week.  Furthermore,  from  Figure  20, 
the  value  of  that  contact  is  about  four  and  a  half  on  a  seven-point  scale. 
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The  substantial  difference  in  the  probabilities  of  contacting  a  fellow  student  and  a 
NPS  staff  member,  less  than  a  third  as  likely,  can  also  be  explained.  As  highlighted  in  the 
previous  section,  it  is  reasonable  to  assume  that  graduates  have  a  greater  number  of  social 
relationships  with  fellow  students  compared  to  NPS  staff.  Furthennore,  the  larger  number 
of  student-peer  relationships  cross  a  diverse  range  of  educational  topics.  Whereas  the  less 
common  NPS  staff  relationships  would  be  expected  along  more  narrowly  defined  topics. 
Therefore,  the  weekly  probability  of  contacting  a  NPS  staff  member  should  naturally  be 
less  than  the  probability  of  contacting  one  of  the  many  student-peers.  This  is  compounded 
by  the  fact  that  many  graduates  work  in  the  same  commands  or  in  common  geographic 
regions. 


F.  SUMMARY  OF  ANALYSIS 

The  analysis  performed  on  the  subjective  data  of  survey  respondents  lead  to 
several  observations. 

•  The  composition  of  the  1ST  curriculum  is  appropriate  to  meet  the  Marine 
Corps’  graduate  level  information  technology  education  requirements. 
However,  it  appears  that  slight  modification  to  the  curriculum  (some  of 
which  are  currently  underway)  could  better  align  the  curriculum  to  the 
requirements. 

•  Officers  selected  to  study  the  1ST  curriculum  have  the  potential  to  use 
their  education  to  advance  the  Marine  Corps  in  areas  other  than  their  9648 
MOS  positions.  Thesis  efforts  can  provide  direct  value  to  the  service  while 
meeting  the  educational  objective  of  the  thesis  process.  Additionally,  the 
1ST  education  is  used  and  presumably  provides  value  in  any  billet 
assignment. 

•  There  appears  to  be  a  relationship  between  students’  interest  in  particular 
topics  and  assessments  of  value  in  those  topics.  This  has  implications  in 
the  methods  that  education  quotas  are  provisioned  and  how  curriculums 
are  administered. 

•  Social  relationships  developed  during  resident  education  provide 
subsequent  value  to  the  organization.  While  not  the  primary  objective  of 
the  education,  this  facet  of  the  resident  education  experience  should  be 
acknowledged  when  comparing  resident  and  non-resident  education 
delivery  methods. 
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IV.  MEASURING  THE  RETURN  ON  EDUCATION 


It  is  not  possible  to  measure  the  exact  monetary  value  of  graduate  education  to  the 
Maine  Corps.  However,  this  chapter  describes  a  theory  and  method  to  estimate  the 
relative  return  on  educational  investment  using  process  output  as  a  surrogate  for  value,  or 
a  numerator,  with  the  cost  to  produce  those  outputs  as  a  denominator  to  form  a  ratio.  The 
method  is  applied  using  actual  data  as  a  proof  of  concept  of  how  to  generate  relatively 
objective  estimates  of  return  on  graduate  education. 

A.  THEORY 

As  indicated  in  this  paper’s  literature  review,  there  are  many  approaches  to 
measuring  the  value  of  education.  Regardless  of  the  approach,  common  units  of  value 
must  be  identified  in  order  to  collect  data  and  make  meaningful  comparisons.  At  a 
national  level,  these  analyzes  have  compared  gross  domestic  product  to  average 
education  levels.  In  corporate  settings,  stock  price  has  been  compared  to  company  costs 
associated  with  employee  learning.  To  analyze  local  training  value,  training  costs  have 
been  compared  to  estimations  of  improved  job  perfonnance.  And  in  non-profit 
organizations,  promotion  rates  have  been  compared  to  education  levels. 

Most  of  these  studies  focused  on  education  as  a  way  to  ensure  a  given  set  of 
capabilities.  While  the  focus  on  capabilities  provided  insights  on  the  value  of  education, 
these  studies  failed  to  show  how  education  affects  subsequent  productivity  in  an 
organization,  based  on  actual  exercise  of  those  capabilities,  presumably  provided  by  the 
education. 

One  way  to  estimate  the  value  of  education  in  practice  would  be  to  measure  both 
the  amount  of  knowledge  used  in  organizational  processes  to  produce  given  outputs  and 
the  cost  to  use  this  knowledge.  Establishing  common  units  of  knowledge  and  a 
relationship  of  knowledge  to  value  would  resolve  the  question  of  how  much  value  the 
NPS  1ST  education  provided  its  students. 

1.  Knowledge  Value  Added 

A  method  called  Knowledge  Value  Added  or  KVA  (Housel  and  Bell  2001)  is 
used  in  this  chapter  to  estimate  more  objectively  the  relationship  between  knowledge  and 
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value  in  sample  processes.  The  KVA  method  reveals  how  knowledge  is  actually  used  at  a 
process  level  in  the  organization.  An  organizational  perspective  is  achieved  by 
aggregating  the  knowledge  contributions  within  the  specific  organizational  processes. 
The  KVA  model  and  its  underlying  assumption  are  presented  in  Figure  22. 

Model:  Change,  Knowledge,  and  Value  are  Proportionate 
Input  Process  Output 

X - <^P  J) - *  Y 

P(X)=Y 

Fundamental  assumptions: 

1 .  If  X=Y,  no  value  has  been  added. 

2.  “Value”  is  proportional  to  “Change” 

3.  “Change”  can  be  measured  by  the  amount  of 
knowledge  required  to  make  the  change. 

So  “value”  is  proportional  to  “change”  is  proportional  to 

“the  amount  of  knowledge  required  to  make  the  change” 

Figure  22.  Fundamental  Assumptions  of  Knowledge  Value  Added  (From:  Fiousel  and  Bell, 

2001) 

Using  the  KVA  approach  to  measure  the  knowledge  embedded  in  processes 
requires  that  common  units  be  established  to  measure  knowledge  across  diverse 
processes.  The  costs  attributed  to  each  process  must  also  be  determined  in  standard  units. 

The  validity  of  the  KVA  method  is  based  on  its  fundamental  assumptions — any 
change  between  an  input  and  an  process  output  must  be  proportional  to  the  value  derived 
from  the  process.  As  for  reliability,  the  results  of  the  KVA  method  are  as  reliable  as  the 
estimations  of  the  measured  processes.  Therefore,  the  KVA  method  is  able  to  increase 
reliability  by  increasing  the  fidelity  of  process  description  and  execution. 

The  KVA  method  of  measuring  knowledge  differs  from  other  approaches.  In 
KVA,  knowledge  is  identified  in  terms  of  processes.  Other  approaches  represent 
knowledge  as  capabilities  waiting  to  be  used.  While  this  is  a  useful  view  of  knowledge  in 
some  circumstances,  it  makes  it  difficult  to  detennine  how  the  knowledge  is  actually  used 
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and  how  the  usage  compares  across  different  processes.  This  knowledge  usage  and 
comparison  is  what  the  KVA  method  provides. 

2.  Measuring  Knowledge  Embedded  in  Processes 

The  crux  of  KVA  is  describing  an  organizational  process  in  a  way  that  represents 
the  knowledge  required  to  perfonn  the  process.  This  description  must  also  be  applicable 
to  any  process  to  allow  objective  comparisons  between  organizational  processes.  To 
measure  the  knowledge  required  to  perform  a  process,  the  process  must  be  described  in 
terms  of  the  changes  that  result  from  the  use  of  knowledge.  There  are  many  ways  to 
perform  this  description  (Housel  and  Bell,  2001);  three  are  explained  here. 

First,  a  process  can  be  described  by  calculating  the  amount  of  time  it  takes  to 
educate  someone  to  successfully  perform  the  process.  This  requires  an  assumption  about 
the  person  being  educated,  a  high  school  graduate,  for  example.  Gathering  this  data 
usually  involves  interviewing  people  experienced  in  the  process.  From  these,  process  and 
sub-process  descriptions  are  recorded  along  with  the  assessments  of  the  learning  time 
required  to  prepare  someone  to  perform  the  process.  Using  learning  time  as  a  surrogate 
for  knowledge  provides  an  objective  means  to  compare  dissimilar  processes. 

A  second  method  to  capture  the  knowledge  embedded  in  a  process  is  to  describe 
the  process  in  tenns  of  instructions.  Similar  to  the  previous  approach,  this  method  records 
the  number  of  instructions  required  to  accomplish  a  particular  process;  processes  that  are 
more  complicated  have  more  instructions  and  thus  more  embedded  knowledge.  While  it 
depends  on  the  process,  collecting  this  data  usually  involves  interviewing  people  skilled 
in  the  process.  With  number  of  instructions  as  a  common  unit,  comparisons  across 
processes  are  possible. 

A  third  approach  extends  from  the  last  and  entails  describing  processes  as  a  series 
of  yes/no  questions.  This  binary  query  method  captures  process  knowledge  by  calculating 
the  number  of  yes/no  questions  required  to  successfully  perform  the  process  (Pickering 
1995).  It  reduces  the  process  to  a  quantity  of  bits.  An  analogy  for  this  approach  is  the 
digital  encoding  of  a  photographic  image.  A  large  or  high-resolution  image  requires  more 
bits  to  encode  than  a  smaller,  low-resolution  image.  The  analogy  continues;  the  better  the 
image,  the  more  bits,  and  the  more  knowledge.  Binary  query  provides  a  way  to  measure 
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knowledge  that  spans  process — any  process  can  be  described  using  this  approach. 
However,  the  actual  application  of  this  approach  is  involved  and  should  be  balanced 
against  the  objectives  of  the  subsequent  calculations. 

3.  Measuring  Process  Cost 

Cost  detennination  is  required  to  complete  the  value-cost  ratios.  Capturing 
process  cost  involves  isolating  the  revenues  allocated  to  accomplish  particular  processes. 
It  is  important  to  use  consistent  process  description  while  collecting  knowledge  and  cost 
data.  There  are  a  number  of  ways  to  gather  cost  data.  Most  organizations  have  a  method 
to  allocate  costs  across  their  business  units.  This  cost  data  can  be  allocated  to  the  process 
level  by  interviewing  managers  familiar  with  the  processes. 

Armed  with  the  amount  of  knowledge,  in  standard  units,  embedded  in  a  process, 
and  the  process  costs,  a  ratio  between  cost  and  value  can  be  created  to  compare  different 
processes  in  common  units. 

B.  METHOD 

The  operationalism  of  the  KVA  method,  to  support  this  research,  is  described  in 
this  section.  This  proof  of  concept  involved  subjects  who  completed  the  Naval 
Postgraduate  School  (NPS)  Information  System  Technology  (1ST)  curriculum  and  relates 
to  processes  that  fonned  their  post-education  jobs.  Data  was  collected  via  a  series  of 
telephone  interviews;  data  collection  for  the  principal  processes  associated  with  a  specific 
job  was  completed  in  less  than  one  hour.  The  specific  implementation  of  the  KVA  theory 
follows. 

1.  Process  Identification 

The  identification  of  processes  begins  with  a  subject’s  job  description.  In  this 
study,  three  or  four  processes  were  described  for  each  information  technology  job 
actually  held  by  the  subjects.  For  each  of  the  job  processes,  subjects  provided  a 
breakdown  of  the  percent  of  their  time  they  spent  on  each  process  during  the  course  of  a 
typical  year.  These  percentages  were  converted  into  days  and  labeled  as  an  occurrence; 
see  Table  14. 
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2.  Value  Measurement 

Knowledge  or  value  was  measured  by  estimating  the  relative  learning  time 
required  to  prepare  a  person  to  perform  one  of  the  identified  processes  (the  first  of  the 
three  process  description  methods  explained  in  the  previous  section).  The  subjects  were 
asked  to  assume  that  the  person  receiving  the  learning  had  no  specific  technical  education 
but  did  understand  the  organization:  a  Marine  officer  of  similar  grade  and  experience  but 
with  no  specialized  technical  education.  The  learning  times  were  normalized  to  one 
hundred  units  to  standardize  subsequent  comparisons.  These  one  hundred  units  of 
learning  time  were  distributed  across  the  identified  processes  for  each  job.  Additionally, 
subjects  were  asked  to  ordinally  rank  the  processes  by  learning  difficultly.  This  served  as 
a  check  of  the  learning  time,  based  on  the  assumption  that  difficult  processes  require 
more  preparation  time  than  simpler  processes. 
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Software  24  5  4 

configuration  release 

30% 

10% 

20% 

40% 

Units  of  Knowledge  Embedded  in  Each  Process 

TOTAL 

Software  requirements 

0 

648 

972 

0 

0 

0 

0 

1296 

324 

3240 

Acquisition  management 

1920 

0 

0 

0 

1440 

480 

960 

0 

0 

4800 

Database  construction 

0 

0 

0 

216 

0 

0 

0 

0 

24 

240 

Software  configuration  release 

36 

0 

12 

0 

24 

0 

0 

48 

0 

120 

Software  Project  Manager 

(Total  for  Job) 

1956 

648 

984 

216 

1464 

480 

960 

1344 

348 

8400 

Table  14.  Knowledge  Value  Calculations 


To  attain  an  additional  level  of  detail,  subjects  were  asked  to  allocate  learning 
time  across  the  education  topics  of  the  1ST  curriculum,  within  each  process.  With  this 
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data,  the  knowledge  associated  with  each  process  was  calculated  by  each  education  topic. 
Table  14  shows  one  example  for  a  Software  Project  Manager.  The  top  half  of  the  table 
consists  of  data  recorded  during  the  interview.  The  bottom  half  shows  the  results  of  the 
calculations  performed  on  the  collected  data. 

In  this  example,  the  job  of  Software  Project  Manger  consisted  of  four  main 
processes:  Software  requirements,  Acquisition  management,  Database  construction,  and 
Software  configuration  release.  The  number  of  days  each  year  that  the  subject  performs 
each  process  is  recorded  under  the  Occurrence  field.  The  Relative  Learning  Time  reveals 
the  relative  complexity  of  each  process  that  the  subject  assessed  for  each;  these  learning 
times  sum  to  one  hundred  units  for  all  processes.  The  ordinal  ranking  of  process 
difficulty,  where  one  represents  the  most  difficult  process  to  learn,  validates  these  values. 
The  data  collection  concluded  with  the  subject’s  assessment  of  how  learning  should  be 
allocated  across  topics  within  each  process.  These  values  are  shown  as  percentages  in  the 
top  half  of  the  table,  and  they  sum  to  one  hundred  percent  for  each  process. 

Using  the  collected  data  from  the  top  half  of  the  table,  the  estimated  number  of 
knowledge  units  is  calculated  for  each  process.  The  calculations  consist  of  multiplying 
the  occurrence  frequency  by  the  relative  learning  time  by  the  percent  allocation  for  each 
topic.  The  resultant  value  is  assigned  to  the  same  topic  and  process.  To  illustrate,  units  of 
Computer  Networking  knowledge  embedded  in  the  Software  Requirements  process  is 
given  by  multiplying  (72)  by  (45)  by  (20%)  to  yield  648.  After  completing  these 
calculations  for  each  topic-process  pair,  the  knowledge  units  are  summed  by  education 
topic  and  by  process. 

These  computations  show  the  relative  quantity  of  knowledge,  allocated  by  topic, 
associated  with  each  process.  For  example,  Acquisition  Management  is  the  highest 
knowledge  process  despite  being  the  second  hardest  process  to  learn.  This  is  because 
Acquisition  Management  is  a  process  that  is  conducted  more  frequently.  Not  surprisingly, 
Acquisition  and  Program  Management  is  the  education  topic  with  the  largest  number  of 
knowledge  units. 
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3.  Cost  Measurement 

Cost  is  computed  by  capturing  the  salary  cost  for  the  person  who  performs  the 
particular  job,  i.e.,  uses  the  knowledge  to  produce  outputs  of  given  processes.  This  salary 
cost  can  then  be  assigned  to  processes  and  sub-processes  based  on  the  percentage  of  time 
that  the  person  engages  in  those  activities.  Furthermore,  costs  can  be  segmented 
according  to  the  education  topics  that  are  required  to  perform  the  processes  and  sub¬ 
processes.  Table  15  continues  the  previous  example  and  demonstrates  the  method  used  to 
collect  and  calculate  costs. 
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$  243.81/ 
Day 

72 

$  17,280 

120 

$  28,800 

24 

$  5760 

24 

$  5760 

Breakdown  of  Topic  Use  within  each  Process 
10%  30%  30%  30% 

20%  14% 


100% 


10%  30%  25%  5%  30% 


Costs  Associated  with  each  Process 

TOTAL 

Software  requirements 

$  1728 

$5184 

$  5184 

$5184 

$17,280 

Acquisition  management  $19,008 

$  5760 

$4032 

$28,800 

Database  construction 

$  5760 

$  5,760 

Software  configuration  release 

$  576 

$  1728 

$1440 

$288 

$  1728 

$  5,760 

Software  Project  Manager 

(Total  for  Job)  $19’008 

$  1,728 

$  5,760 

$  5,760 

$  1,728 

$  1,440 

$  6,048 

$10,944 

$  5,184 

$57,600 

Table  15.  Sample  Process  Cost  Calculations 


The  data  in  the  top  half  of  Table  15  was  gathered  from  the  subject  during  the 
interview.  It  consists  of  the  same  processes  and  occurrence  rates  described  in  the  previous 
table.  Within  each  of  these  processes,  the  subject  provided  an  assessment  of  how  much 
his  knowledge  of  the  education  topics  was  actually  used  during  the  execution  of  the 
processes.  This  is  recorded  as  the  percentage  values  in  the  top  half  of  the  table.  Lastly,  a 
daily  salary  rate  was  calculated  based  on  the  rank  and  experience  of  the  subject. 
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The  collected  data  is  used  to  calculate  the  costs  related  to  performing  the  various 
processes.  The  yearly  cost  of  each  process  is  computed  by  multiplying  the  occurrence 
rate  by  the  daily  salary  rate.  The  sum  of  these  costs  equals  the  subject’s  total  yearly 
salary.  The  costs  associated  with  each  topic  and  process  is  calculated  by  multiplying  the 
yearly  cost  of  the  process  by  the  percentage  of  time  that  each  topic  was  used.  Lastly,  the 
costs  are  summed  by  topic  and  by  process. 

4.  Comparisons 

With  every  processes  described  in  common  units  of  knowledge  and  ascribed  costs 
based  on  the  use  of  the  knowledge,  value  comparisons  can  be  performed  between  topics, 
processes,  and  even  between  subjects.  To  demonstrate  some  of  the  possible  comparisons, 
Table  16  lists  the  value  ratios  from  the  three  subjects  examined  in  this  study.  While 
comparisons  could  be  perfonned  at  the  process  level,  Table  16  provides  only  the 
aggregation  of  multiple  processes  in  the  form  of  three  different  jobs. 

The  table  displays  value  in  two  fonns:  return  on  knowledge  and  cost  per  unit  of 
knowledge.  The  return  on  knowledge  percentage  is  derived  from  dividing  job  output 
(knowledge)  by  cost.  A  hurdle  rate — return  on  knowledge  rate  for  the  job — for  each  job 
is  also  computed  from  the  total  knowledge  divided  by  the  total  job  cost.  From  this 
information,  many  types  of  comparisons  are  possible. 

Before  presenting  example  comparisons,  it  is  important  to  caveat  the  fidelity  of 
these  sample  calculations.  Wide  ranging  conclusions  should  not  be  drawn  from  these 
particular  calculations  due  to  the  small  and  limited  sampling  used  to  support  them. 
However,  these  calculations  do  support  rough-cut  estimations  and  reveal  order  of 
magnitude  relationships  that  may  not  be  otherwise  obvious.  Moreover,  the  time  costs 
associated  with  preparing  this  information  is  about  an  hour  per  job. 

The  ratios  in  Table  16  reveal  several  interesting  relationships.  Starting  with  the 
hurdle  rates,  it  is  evident  that  some  education  topics  achieve  higher  returns  than  others.  In 
this  case,  the  Managing  Change  and  Professional  Military  Education  topics  surpass  the 
hurdle  rate  for  all  three  job  descriptions.  This  suggests  that  these  topics  are  frequently 
employed  in  these  jobs.  Conversely,  several  topics  fail  to  reach  the  hurdle  rate  in  any  of 
the  jobs,  which  suggests  that  they  are  used  infrequently.  These  comparisons  are  helpful  to 
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identify  trends  in  education  topics  across  multiple  jobs  and  processes,  because  the  jobs 
and  processes  are  described  in  common  units. 


Acquisition  and 

Program 

Management 

Command  and 

Control 

Communication 

System 

Engineering 

Computer 

Networking 

Computer  and 

Network  Security 

Database 

Economic  and 

Financial 

Management 

Managing 

Change 

Professional 

Military  Education 

Software 

Engineering 

Software 

Language 

Programming 

Output/Cost 

Hurdle 

Rate 

Software  Project 
Manager 

14.6% 

10.3% 

37.5% 

17.1% 

3.8% 

84.7% 

33.3% 

15.9% 

12.3% 

6.7% 

Executive  Officer 

12.7% 

49.4% 

3.1% 

1.6% 

19.4% 

14.4% 

6.1% 

Develop  Digital 
Massage  Standards 

14.7% 

18.4% 

8.1% 

8.1% 

8.5% 

18.3% 

16.2% 

Cost/Unit  of  Knowledge 

Software  Project  Manager 

$9.72 

$2.67 

$  5.85 

$  26.67 

$  1.18 

$3.00 

$6.30 

$8.14 

$  14.90 

Executive  Officer 

$2.02 

$  32.00 

$  64.00 

$  5.14 

$  6.95 

$16.53 

Develop  Digital  Massage 
Standards 

$5.44 

$12.36 

$  12.36 

$  1 1 .74 

$  5.46 

$6.18 

Table  16.  Cost-Knowledge  Ratios 


These  ratios  can  also  highlight  inconsistencies  between  jobs  and  processes.  As  an 
example,  the  cost  per  unit  of  Economic  and  Financial  Management  knowledge  between 
the  Software  Project  Manager  and  the  Executive  Officer  jobs  differ  by  much  more  than 
an  order  of  magnitude.  While  some  of  this  difference  likely  stems  from  the  inaccuracies 
of  the  data  sampling,  such  a  large  difference  suggests  that  the  Software  Project  Manager 
uses  his  knowledge  in  this  topic  much  more  frequently  than  the  Executive  Office,  which 
drives  down  the  unit  cost  of  the  knowledge. 

A  final  type  of  comparison,  not  demonstrated  in  this  example,  can  be  performed 
for  different  people  performing  the  same  processes.  Once  the  knowledge  embedded  in  a 
process  is  estimated,  the  only  way  to  change  the  value  ratio  is  by  altering  the  costs 
required  to  perfonn  the  process.  This  can  be  witnessed  in  two  ways:  change  the  employee 
costs  associated  with  the  process  and/or  change  the  frequency  that  the  process  is 
completed. 
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This  effect  can  be  shown  by  a  simple  example.  For  a  process  such  as  sweeping  the 
floor,  there  is  an  established  amount  of  knowledge  required.  Regardless  who  performs  the 
task,  the  same  amount  of  knowledge  is  expended.  This  is  the  fundamental  premise. 
However,  in  order  to  identify  the  most  effective  sweeper,  sweeper  cost  and  speed  must  be 
considered.  For  example,  if  Sweeper  A  gets  paid  twice  as  much  as  Sweeper  B 
(presumable  due  to  a  greater  knowledge  capability)  but  sweeps  three  times  faster  than 
Sweeper  B,  then  Sweeper  A  has  a  lower  cost  per  knowledge  use. 

This  is  a  fundamental  distinction  of  the  KVA  method.  It  separates  KVA  from 
other  knowledge  valuation  models  such  as  Phillips’  return  on  investment  process  for 
training.  Most  processes  compare  the  costs  of  knowledge  acquisition  or  capability  to  a 
subjective  estimation  of  value  that  results  from  the  acquired  knowledge.  In  KVA,  the 
knowledge  acquisition  costs  are  only  relevant  to  the  extent  that  they  influence  the  salary 
cost  of  the  employee.  Knowledge  value  added  computations  focus  on  representing 
processes  in  common  units  and  allocating  a  cost  for  the  completion  of  each  process. 
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V.  DISCUSSION 


A.  RESEARCH  QUESTION  SUMMARY  AND  IMPLICATIONS 

The  results  of  this  study  are  best  described  by  answering  each  of  the  specific 

research  questions. 

1.  Meeting  Requirements 

How  well  does  the  Naval  Postgraduate  School  (NPS)  Information  System 
Technology  (1ST)  curriculum  meet  Marine  Corps  graduate  level  Information  Technology 
(IT)  education  requirements?  Analysis  indicates  that  the  1ST  curriculum  satisfactorily 
meets  the  Marine  Corps’  graduate  level  IT  education  requirements.  Figure  8  and  Figure 
10  show  that  the  majority  of  the  Marine  Corps’  education  requirements,  described  by 
Billet  Education  Evaluation  Certificates  (BEEC)  and  graduate  responses,  are  included  in 
the  1ST  curriculum. 

Moreover,  Figure  9  and  Figure  1 1  compare  the  relative  weight  of  requirements  to 
the  curriculum  emphasis  of  topics  to  show  that  the  time  devoted  to  most  topics  is  well 
balanced  and  evolving  to  meet  requirements.  The  evolutionary  nature  of  the  1ST 
curriculum  content  indicates  that  the  NPS  is  proactively  engaged  in  providing  education 
to  meet  the  changing  information  technology  environment  in  the  Department  of  Defense. 

2.  Curriculum  Value 

What  is  the  value  of  the  1ST  education  to  the  Marine  Corps?  This  study  provides 
several  indications  that  the  1ST  education  provides  significant  value  to  the  Marine  Corps. 
First,  the  education  meets  the  sponsor’s  requirements,  which  presumably  indicates  that 
results  of  the  education  are  worth  their  costs.  Second,  1ST  curriculum  enrollment  of 
Marine  officers  has  grown  over  the  past  ten  years.  Invariably,  one  source  of  this  increase 
is  the  growing  importance  of  infonnation  technology  within  the  service.  However,  the 
specific  decision  to  send  additional  officers  to  participate  in  the  1ST  curriculum  is  a 
strong  sign  of  Maine  Corps  satisfaction  and  perceived  return  on  investment  from  the 
program. 

Lastly,  Marine  graduates  with  this  education  indicate  that  they  are  able  to 
successfully  influence  Marine  Corps  processes  as  a  result  of  their  1ST  education.  Using  a 
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seven-point  scale  to  assess  value  (one=low,  seven=high),  graduates  responded  that  the 
average  value  of  the  1ST  curriculum,  across  all  topics,  was  4.4  in  IT  billets  and  4.0  in  all 
other  billets.  These  ratings  indicate  relatively  high  curriculum  value  and  are  reinforced  by 
the  data  in  Figure  14  and  Figure  15.  These  figures  show  graduates’  assessment  of  the 
value  of  knowledge  (mean=5.8)  and  skills  (mean=5.2)  acquired  at  NPS.  Furthennore, 
Figure  16  reveals  that  over  sixty  percent  of  survey  respondents  say  that  they  use  their 
NPS  acquired  knowledge  and  skill  on  a  daily  basis.  These  findings  suggest,  quite 
dramatically,  that  graduates  of  the  1ST  curriculum  believe  that  their  NPS  education 
increases  their  ability  to  positively  influence  the  Marine  Corps. 

3.  Curriculum  Impact  on  Productivity 

How  does  the  NPS  1ST  curriculum  improve  productivity  within  Marine  Corps 
SEP  billets?  This  study  identified  several  ways  that  the  1ST  education  improves 
productivity  in  the  Marine  Corps.  These  improvements  take  form  in  the  individual 
actions  of  Marines  who  completed  the  education,  as  well  as,  from  tools  produced  by  these 
Marines,  that  in  turn  improve  the  productivity  of  Marines  throughout  the  Corps. 

The  most  common  way  that  the  1ST  education  improves  productivity  in  the 
Marine  Corps  is  by  allowing  graduates  to  “hit  the  ground  running”  when  they  enter  their 
information  technology  billets.  They  report  that  they  are  knowledgeable  and  skilled  in 
technical  issues  relating  to  their  responsibilities.  While  most  survey  respondents  felt  that 
their  education  helped  them  transition  into  IT  billets,  several  also  highlighted  the  ability 
to  make  better  decisions  as  a  result  of  their  education.  Additionally,  relationships 
developed  with  classmates  and  NPS  staff  members  appear  to  increase  productivity  in 
many  cases. 

The  thesis  research  perfonned  by  1ST  students  can  also  improve  productivity.  In 
one  case,  a  scheduling  tool  was  developed  and  fielded  to  Marine  Corps  recruiters.  It  is 
estimated  that  the  tool  saved  each  recruiter  sixty  hours  a  year.  In  several  other  cases, 
students  were  able  to  perform  various  studies  and  analyses  that  would  have  otherwise 
been  completed  by  a  Marine  not  in  a  student  status. 
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What  these  productivity  improvements  have  in  common  is  that  the  Marine  Corps 
can  reap  greater  benefits  from  its  1ST  officer  students  by  ensuring  that  they  remained 
focused  on  solving  Marine  Corps  problems. 

4.  Personal  Desires  and  Value 

What  effect  do  personal  desires  have  on  education  value?  This  research  has 
shown  some  positive  correlation  between  personal  education  desires  and  assessments  of 
education  value.  This  suggests  that  greater  organizational  value  will  be  realized  when 
students  are  provided  options  to  fulfill  their  personal  education  aspirations.  There  are  two 
areas  where  this  is  relevant. 

The  first  time  that  personal  and  service  educational  objectives  come  into  potential 
conflict  is  during  the  Marine  Corps  screening  and  selection  of  Special  Education  Program 
(SEP)  officers.  Applicants  list  their  curriculum  choices  by  priority.  However,  the  “needs 
of  the  Corps”  often  take  precedence  over  individual  desires.  While  these  needs  must  be 
met  and  everyone  is  not  likely  to  get  their  first  choice,  understanding  the  potential  payoff 
of  attaining  assignment  satisfaction  is  worthwhile.  The  current  curriculum  assignment 
process  is  so  focused  on  meeting  the  Corps’  needs  that  less  popular  curriculums  are  filled 
by  the  first  suitable  applicant,  regardless  of  applicant  desires  (Esparza  2002b). 

The  second  time  that  students  have  the  option  to  tailor  their  education  is  during 
the  execution  of  the  curriculum.  In  the  case  on  the  1ST  curriculum,  the  default  course  list 
is  evolving.  Where  previously,  students  where  given  only  two  course  electives;  the 
curriculum  is  beginning  to  institute  topic  “tracks”  that  allow  students  to  select  one  of 
several  three-course  series.  These  tracks  provide  for  specialization  beyond  the  default 
curriculum,  which  still  meets  all  sponsor  requirements,  but  fall  short  of  the  time  and 
study  required  for  a  dual  degree. 

Some  may  argue  that  this  small  amount  of  specialization  is  not  worth  the 
additional  study  time,  especially  if  the  base  courses  meet  sponsor  expectations.  However, 
this  study  suggests  that  the  extra  two  or  three  courses  that  constitute  the  track  may  be 
worth  the  extra  curriculum  time.  First,  the  research  presented  in  this  paper  indicates  that 
student-selected  tracking  results  in  higher  perceived  value.  Additionally,  the  educational 
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focus  provided  by  a  course  track  is  likely  to  align  with  thesis  research,  which  should 
benefit  fonn  the  additional  depth  of  study. 

5.  Social  Relationships  and  Value 

Is  there  a  relationship  between  social  connections  formed  during  graduate 
education  and  subsequent  productivity?  This  research  indicates  that  social  relationships 
are  established  and  maintained  because  of  resident  education.  The  value  of  these 
relationships  is  found  during  subsequent  billet  responsibilities.  This  was  most 
dramatically  demonstrated  in  Figure  20  where  the  average  value  of  an  NPS  relationship 
was  4.5  (on  a  zero  to  seven  point  scale)  and  Figure  21  where  the  average  probability  of 
calling  on  that  relationship  during  a  given  week  was  fifty-eight  percent. 

Clearly,  these  social  aspects  of  education  are  less  important  than  the  knowledge 
and  skills  developed  at  the  NPS.  However,  these  relationships  are  a  byproduct  of  the 
current  process,  and  this  study  shows  that  they  do  contribute  substantially  to  subsequent 
efforts.  Understanding  how  certain  social  connections  provide  value  is  important  to 
increasing  the  potential  of  education  provided  in  the  current  resident  process,  as  well  as, 
to  designing  distant  education  approaches  that  appreciate  and  foster  student  relationships. 

6.  Measuring  the  Value  of  Education 

Can  a  Knowledge  Value  Added  (KVA)  method  be  used  to  assess  the  current  value 
of  graduate  education?  The  KVA  method  can  be  used  to  measure  and  compare  the  value 
of  education.  The  approach  used  in  this  study  was  able  to  reveal  the  relative  contributions 
of  different  education  topics  to  different  processes  within  the  examined  jobs.  The  results 
of  the  proof  of  concept,  explained  in  Chapter  Four,  suggests  that  a  similar  KVA  method 
could  be  used  in  the  SEP  to  make  decisions  that  are  more  infonned. 

In  an  organizational  context,  education  produces  capabilities.  Most  studies  of 
educational  value  within  organizations  focus  on  describing  and  estimating  the  value  of 
these  capabilities.  With  the  exception  of  the  Phillips’  Return  on  Investment  approach, 
most  fail  to  show  how  the  capabilities  are  used  in  the  organization.  The  KVA  method 
steps  beyond  Phillips’  by  providing  a  method  to  show  and  compare  education  use  in 
common  units. 
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The  ability  to  see  education  (capability)  in  use  is  a  central  theme  of  the  KVA 
method.  A  capability  that  is  not  used  remains  merely  an  option.  However,  when  the 
option  is  exercised  to  provide  some  productivity,  it  yields  value,  which  can  be  measured 
and  assessed  using  a  KVA  method  (Housel  2002). 

B.  RECOMMENDATIONS 

The  benefit  of  this  research  can  be  viewed  as  contributing  to  three  domains:  the 
Marine  Corps,  the  Naval  Postgraduate  School,  and  the  application  of  KVA  to  measure 
education. 

1.  Consumers 

In  the  context  of  this  research,  the  United  States  Marine  Corps  are  the  consumers 
of  graduate  level  education.  The  SEP  in  the  Marine  Corps’  program  designed  to  manage 
the  service’s  graduate  education  investments.  In  general,  the  SEP  fulfills  the  mandates  of 
its  charter:  requirements  are  adequately  identified  and  provisioned.  However,  close 
examination  of  the  program  reveals  potential  opportunities  to  improve  the  overall 
performance  of  the  program  to  meet  organizational  needs. 

a.  Advocacy 

The  operational  and  supporting  commands  of  the  Marine  Corps  fail  to 
leverage  the  educational  and  research  opportunities  provided  by  students  attending  the 
NPS.  Every  Marine  student  assigned  to  the  NPS  has  several  years  of  experience  in  one  of 
the  Marine  Corps’  primary  occupational  specialties.  However,  once  assigned  to  NPS,  the 
connection  between  the  student  and  his  occupational  community  is  effectively  severed.  In 
general,  only  student  initiated  actions  cause  the  connection  to  continue. 

The  result  of  this  condition  is  a  lost  opportunity.  Instead  of  building  upon 
their  experience  and  knowledge  to  study  issues  important  to  their  respective 
communities,  many  students  immerse  themselves  in  the  academic  pursuit  of  knowledge 
with  no  particular  plan  to  apply  that  knowledge  to  a  Marine  Corps  subject. 

It  appears  that  this  condition  may  results  from  low  density  of  NPS 
graduates  in  the  Marine  Corps.  Without  a  personal  knowledge  of  the  NPS,  there  is  a 
hesitancy  to  engage  what  is  otherwise  a  foreign  institution. 


73 


To  begin  countering  this  misperception,  a  broker  is  required  to 
communicate  the  issues  that  are  important  to  the  Marine  Corps  and  the  topics  that  are  part 
of  the  various  NPS  curricula.  There  are  several  places  to  look  for  these  brokers.  On  the 
NPS  side,  all  Marine  staff  and  students  should  seek  to  remain  engaged  in  Marine  Corps 
issues.  There  are  examples  in  other  curriculums  where  Marine  instructors  fulfill  this 
function.  However,  the  1ST  curriculum  has  no  Marine  instructors.  Furthermore,  the  1ST 
curriculum  has  the  highest  density  of  Marine  students. 

Outside  of  specific  curriculums,  a  broker  could  be  assigned  to  the  School’s 
Executive  Director  of  Institutional  Advancement  to  formally  voice  Marine  Corps 
concerns  throughout  the  School.  One  of  the  tasks  of  this  office  is  to  ensure  that  the  NPS 
lives  up  to  its  claim  of  Corporate  University  for  the  Navy  and  Marine  Corps. 

On  the  Marine  Corps  side,  occupation  field  managers,  community 
advocates,  and  commands  assigned  SEP  officers  are  in  suitable  positions  to  voice 
educational  and  research  concerns.  These  Marines  often  face  challenges  that  they  are 
unable  to  devote  adequate  resources  towards  solving.  These  challenges  are  exactly  the 
types  of  projects  and  research  that  the  student-officers  at  NPS  could  use  their  experience 
and  education  to  solve,  at  no  additional  cost  to  the  service.  Furthermore,  in  the  process  of 
analyzing  these  problems,  the  student  prepares  himself  for  return  to  a  Marine  Corps 
command  where  he  is  better  able  to  make  a  fast  transition  towards  improving 
productivity. 

b.  Requirements 

The  current  process  of  using  BEEC  to  describe  graduate  education 
requirements  is  flawed.  There  are  two  reasons  why  the  products  of  this  type  of 
requirement  process  inaccurately  represent  the  Marine  Corps’  graduate  education 
requirements.  First,  officers  with  the  designated  education  are  not  necessarily  assigned  to 
the  billets  certified  as  requiring  the  education.  Secondly,  data  provided  in  the  BEEC  is 
too  subjective  to  enable  objective  decisions  to  be  made  concerning  the  allocation  of 
educational  resource. 

Taking  steps  to  solve  these  problems  need  not  require  a  complete  revision 
of  SEP  management  processes.  However,  if  the  Marine  Corps  seeks  to  improve  the 
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performance  of  it  educational  investments,  some  changes  are  warranted.  Currently,  NPS 
graduates  are  assigned  to  commands  that  rate  officers  with  various  NPS  degrees,  based 
upon  a  validated  BEEC.  In  practice,  many  graduates  never  actually  serve  in  the  billet 
described  by  the  BEEC.  In  these  cases,  the  BEEC  serves  only  as  an  administrative  hurdle 
to  gaining  a  SEP  officer.  However,  this  condition  is  not  always  bad.  It  provides  the 
command  leadership  flexibility  to  respond  to  current  situations.  This  flexibility  would  be 
removed  if  SEP  officers  were  rigidly  assigned  to  BEEC-approved  positions. 

While  there  are  positive  and  negative  aspects  of  the  current  practice  of 
assigning  SEP  officer,  the  current  practice  is  not  consistent  with  the  intention  of  the  SEP 
policy  on  billet  education  certification.  Perhaps  a  more  realistic  method  to  assign  SEP 
officers,  and  retain  local  command  flexibility,  would  be  to  assign  NPS  graduates  to 
commands  as  opposed  to  billets.  This  action  would  align  the  policy  with  the  current 
practice.  The  resultant  requirements  process  would  cause  command  to  describe  their 
aggregate  graduate  education  requirements  for  competition  between  other  commands. 

Unfortunately,  this  new  requirement  process  would  remain  as  subjective 
as  the  current  BEEC  process.  Therefore,  to  improve  the  efficiency  and  efficacy  of  service 
investments  in  graduate  education,  a  more  objective  method  of  identifying  educational 
requirements  is  warranted.  This  research  has  advanced  one  such  method,  the  KVA 
method.  While  the  KVA  method  may  not  be  the  specific  approach  used  to  solve  this 
problem,  without  adjusting  the  process  to  increase  the  objectivity,  assessments  of 
program  performance  will  remain  subjective. 

2.  Providers 

In  the  context  of  this  research,  the  Naval  Postgraduate  School  serves  as  the 
graduate  education  provider.  Beyond  the  observation  that  the  NPS  1ST  curriculum  fulfills 
the  sponsor  requirements,  this  research  highlights  two  areas  of  curriculum  management 
that  should  not  be  overlooked.  The  results  of  this  thesis  suggest  that  there  are  important 
curriculum  design  and  feedback  monitoring  decisions  that  are  influential  to  improving  the 
performance  of  graduating  officers. 

Analysis  in  this  study  concluded  that  social  relationships  developed  during 
resident  NPS  education  and  curriculum  latitude  to  satisfy  personal  educational  desires 
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provide  downstream  value.  Two  implications  for  curriculum  designers  emerge  from  these 
conclusions.  The  first  issue  relates  to  efforts  to  provide  graduate  education  in  non¬ 
resident  formats.  While  the  objective  of  non-resident  or  distant  learning  education  is  to 
provide  education  at  a  quality  similar  to  the  resident  experience,  this  research  indicates 
that  this  objective  will  not  be  attained  without  accounting  for  the  social  interactions  of 
staff  and  students. 

A  second  issue,  curriculum  design  consideration  relates  to  meeting  the 
educational  desires  of  the  student  as  well  and  the  curriculum  sponsors.  The  ongoing 
efforts  by  the  NPS  Department  of  Information  Sciences  to  provide  curriculum  tracking 
seems  to  be  an  appropriate  approach  to  achieving  this  balance. 

This  research  also  advanced  an  approach  to  improve  curriculum  feedback  beyond 
the  current  Education  Skill  Requirement  process.  One  advantage  of  the  KVA  method  of 
measuring  knowledge  is  its  ability  to  show  performance  changes.  Because  KVA  uses  a 
fixed  description  of  a  process,  any  changes  that  increase  (or  decrease)  overall  process 
performances  are  attributable  to  the  person  performing  the  process.  Therefore,  it  the 
person  perfonning  the  process  increases  their  knowledge  or  skills,  process  performance 
will  also  increase  and  be  captured  during  the  KVA  computations.  Furthermore, 
educational  changes  can  be  compared  across  jobs  and  processes  because  KVA  describes 
all  jobs  and  processes  in  common  units. 

3.  Feedback 

This  study  described  and  demonstrated  a  method  to  estimate  the  value  of 
education  in  the  context  of  the  Marine  officers  who  have  been  educated  in  the  1ST 
curriculum  of  the  NPS.  The  proof  of  concept  revealed  one  way  in  which  KVA  could  be 
used  to  detennine  and  compare  the  value  of  education.  However,  the  method 
demonstrated  in  this  proof  of  concept  is  not  limited  to  the  context  of  this  paper. 
Hopefully,  this  proof  of  concept  has  revealed  the  potential  value  of  the  KVA  method  in 
different  situations. 

The  KVA  approach  does  not  try  to  calculate  the  knowledge  capability  in  an 
organization  by  looking  at  the  employees.  It  detennines  the  quantity  of  organizational 
knowledge  by  examining  the  changes  that  the  knowledge  produces  in  organizational 
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processes.  These  changes  in  output  are  represented  by  the  knowledge  required  to  make 
them.  This  model  is  more  in  keeping  with  the  information  age  notion  that  knowledge  is 
proportionate  to  value,  as  opposed  to  industrial  age  thinking  where  the  specific 
knowledge  capabilities  required  by  the  organization  are  acquired  and  released  based  on 
the  changing  demands  of  the  organization.  This  paradigm  suggests  that  the  half-life  of 
knowledge  is  tied  to  changes  in  process  design. 

The  infonnation  age  approach  to  knowledge  suggests  that  the  half-life  of 
knowledge  depends  on  its  use  (Cook  2002b).  In  this  paradigm,  the  value  of  knowledge 
actually  increases  with  use  because  the  user  employs  the  knowledge  faster  and  to  better 
advantage.  In  this  sense,  knowledge  is  not  a  scarce  resource.  Furthermore,  knowledge 
creation  actually  takes  place  during  use.  As  situations  change,  knowledge  is  adapted  to 
meet  new  requirements  and  new  learning  build  upon  previous  knowledge.  The  KVA 
method  is  able  to  capture  this  feedback  loop  because  expanded  knowledge  allows  the 
user  to  complete  processes  quicker  or  more  often. 
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APPENDIX  A.  BILLET  EDUCATION  EVALUATION 

CERTIFICATE 


BILLET  EDUCATION  EVALUATION  CERTIFICATE  (1500) 

NAVMC1 1 345  (5-97)  Ref:  MCO  P531 1 . 1 C 

SN:  0109-LF-069-0100 

Complete  a  separate  Certificate  for  EACH  billet 

T/O  NUMBER  T/O  LINE  BILLET  TITLE/ORGANIZATION  LOCATION 

NUMBER 

GRADE  MOS  REQUIRED  BY  F/Y  OSD/JCS  POSITION  DESCRIPTION  NO. 


CURRICULUM  SHOULD  INCLUDE  (List  comprehensive  area  and/or  electives  required  for  this  billet) 


JUSTIFICATION  (Simple,  brief  narrative  explaining  why  the  occupant  of  this  billet  requires  the  special  education  described.  What  does  he  do  with  the 
knowledge?  How  does  he  employ  it?) 


SPECIAL  EDUCATION  DISCIPLINE  (Select  the  course  of  instruction  and  alternate  to  be  attended  which  best  fits  RECOMMENDED  LENGTH  (years) 
the  exact  requirements  of  the  billet.) 


RECOMMENDED  INSTITUTION  AND  LOCATION  (Select  school  and  location  best  suited  for  the  exact  EDUCATION  PREREQUISITE 
requirements  of  this  billet.  List  alternate  institution,  if  applicable.)  (MCO  1500.5) 

□  NECESSARY 

□  DESIRABLE 

SIGNATURE  OF  ADVOCATE  or  OCCFLD  Manager/CODE/DATE  SIGNATURE  OF  MOS  Specialist/CODE/DATE 

CG  MCCDC  DECISION 

APPROVED:  DATE  DISAPPROVED:  DATE 


FOOTNOTE: 
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APPENDIX  B.  DEFAULT  MARINE  INFORMATION  SYSTEM 
TECHNOLOGY  COURSE  LIST 


1st 

QTR 

Fall/Spr 

Visual  Basic 

IS-2020 

Computer  Arch 
&  Operating 
Systems 
CS-3030 

Statistics  for 
Technical 
Management 
OS-3105 

Strategy  & 
Policy 

NW-3230 

2nd 

QTR 

Win/Sum 

Software  Design 

IS-3020 

Database 

IS-3201 

Computer 

Networks 

IS-3502 

Ops  Research 
for  Computer 
Systems 
OS-3004 

3rd 

QTR 

Fall/Spr 

Introduction  to 
C4I 

CC-3000 

Decision 

Support 

Systems 

IS-3301 

Principles  of 
Information 
Operations 
IW-3101 

Math  for  Info 
Systems 
Security  Officer 
M  0-1 901 

4th 

QTR 

Win/Sum 

Intro  to  Comm 
System 
Engineering  1 
EO-2514 

Space 

Technology 

Applications 

SS-3011 

Introduction  to 
Computer 
Security 
CS-3600 

C4ISR  System 
Evaluation 

IS-3172 

Managing 
Planned  Change 
in  Complex  Org 
MN-4125 

5th 

QTR 

Fall/Spr 

Intro  to  Comm 
Systems 
Engineering  II 
EO-3514 

Software 
Engineering  & 
Management 
IS-4300 

Information 

Systems 

Evaluation 

IS-4031 

Military  Satellite 
Communications 

SS-3613 

Introduction  to 
Thesis  Research 

IS-3333 

6th 

QTR 

Win/Sum 

Communication 
System  Analysis 

EO-4514 

Thesis  Research 

IS-0810 

Architecting 

Information 

Systems 

IS-4220 

Elective 

7th 

QTR 

Fall/Spr 

Thesis  Research 

IS-0810 

Financial  Mgt  in 
the  Armed 
Forces 
MN-3154 

C4ISR  Systems 

CC-4221 

Principles  of 
Acquisition 
Management 
MN-3331 

8th 

QTR 

Win/Sum 

Thesis  Research 

IS-0810 

Information 

Systems 

Management 

IS-4182 

Thesis  Research 

IS-0810 

Elective 
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APPENDIX  C.  MARINE  CORPS  INFORMATION  TECHNOLOGY 
BILLETS  (AFTER:  KLINGER  2002B) 


Billet  Title 

Command 

Rank 

Owner 

D215/040DISN  JT  PROJ  OFF 

DISN  JOINT  SYSTEMS  OFCR 

CAPT 

PP&O 

DATA  COMMUNICATIONS  OFF 

DFAS/SABRS  KANSAS  CITY 

MAJ 

P&R 

DATA  COMMUNICATIONS  OFF 

DFAS/SABRS  KANSAS  CITY 

MAJ 

P&R 

DATA  COMMUNICATION  OFF 

DFAS/SABRS  KANSAS  CITY 

CAPT 

P&R 

DATA  COMMUNICATION  OFF 

DFAS/SABRS  KANSAS  CITY 

CAPT 

P&R 

DATA  COMMUNICATIONS  OFF 

DFAS/SABRS  KANSAS  CITY 

CAPT 

P&R 

631/06  C4S  STAFF  OFFICER 

HQ  US  TRANSPORT  CMD  JT 

MAJ 

PP&O 

663/03  DATABASE  ENGINEER 

HQ  US  TRANSPORT  CMD  JT 

MAJ 

PP&O 

DIR  INFO  TECH 

MARCORSYSCOM  QUANTICO  VA 

LTCOL 

MCSC 

INFO  TECH  OFFICER 

MARCORSYSCOM  QUANTICO  VA 

MAJ 

MCSC 

DATA  SYSTEMS  MGR 

MARCORSYSCOM  QUANTICO  VA 

CAPT 

MCSC 

MGMT  DATA  SYS  OFFICER 

MARCORSYSCOM  QUANTICO  VA 

LTCOL 

MCSC 

MGMT  DATA  SYS  OFFICER 

MARCORSYSCOM  QUANTICO  VA 

CAPT 

MCSC 

APML 

MARCORSYSCOM  QUANTICO  VA 

MAJ 

MCSC 

MGMT  DATA  SYS  OFC 

MARCORSYSCOM  QUANTICO  VA 

CAPT 

MCSC 

MGMT  DATA  SYS  OFC 

MARCORSYSCOM  QUANTICO  VA 

CAPT 

MCSC 

APM  IS 

MARCORSYSCOM  QUANTICO  VA 

MAJ 

MCSC 

SYSTEMS  ANALYST 

MARCORSYSCOM  QUANTICO  VA 

MAJ 

MCSC 

SYSTEMS  ANALYST 

MARCORSYSCOM  QUANTICO  VA 

MAJ 

MCSC 

HEAD  MISB 

MARCORSYSCOM  QUANTICO  VA 

MAJ 

MCSC 

SYSTEM  ANALYST 

MARCORSYSCOM  QUANTICO  VA 

MAJ 

MCSC 

SYSTEM  ANALYST 

MARCORSYSCOM  QUANTICO  VA 

MAJ 

MCSC 

OPERATIONS  OFF 

COMBINED  MCTSSA  AT  CAMPEN 

LTCOL 

MCSC 

DEPUTY  DIV  DIR 

COMBINED  MCTSSA  AT  CAMPEN 

MAJ 

MCSC 

GND  C2/CSS  PO 

COMBINED  MCTSSA  AT  CAMPEN 

MAJ 

MCSC 

HEAD 

M&RA  HQMC  (OFFICERS) 

LTCOL 

M&RA 

HEAD 

M&RA  HQMC  (OFFICERS) 

LTCOL 

M&RA 

SYSTEMS  ANALYST 

M&RA  HQMC  (OFFICERS) 

MAJ 

M&RA 

DIRECTOR 

M&RA  HQMC  (OFFICERS) 

LTCOL 

M&RA 

SYSTEMS  ANAL 

M&RA  HQMC  MOB  BILLETS 

CAPT 

M&RA 

HEAD,  CPN 

C4  DEPARTMENT  HQMC 

MAJ 

C4 

SATELLITE  COMM  OFFICER 

C4  DEPARTMENT  HQMC 

MAJ 

C4 

INFO  TECH  MGMT  OFFICER 

C4  DEPARTMENT  HQMC 

MAJ 

C4 

SECTION  HEAD 

PP&O  HQMC  MGT  HQ  (OFFICR) 

LTCOL 

PP&O 

DIRECTOR 

MCSA  MISSA 

LTCOL 

PP&O 

CHIEF 

MCI  CO  MB  WASHDC 

MAJ 

TECOM 

SYSTEMS  INTEGRATION  OFF 

PERS  MGT  DIV  HQMC  (MM) 

MAJ 

M&RA 

HEAD,  RECORDS  SECTION 

PERS  MGT  SPT  BR  MCCDC 

MAJ 

MCCDC 

SYSTEMS  OFFICER 

HQ,MC  RCTG  CMD  -  MCRC 

MAJ 

M&RA 

MODELING  &  SIMULATION 

HQ  CO  LFTCPAC 

CAPT 

TECOM 

MAT  MGMT  OFFICER 

MARCORLOGBASES 

MAJ 

MATCOM 

OP  ARCHITECTURE  OFF 

CG  MCCDC 

MAJ 

MCCDC 

MODEL  SPT/DESIGN  OFF 

MAGTF  STF  TRNG  PROG  CNTR 

CAPT 

TECOM 

COMM  OFFICER/ISMO 

MARINE  CORPS  WAR  LAB 

CAPT 

MCCDC 

HEAD,  TRAINING  READINESS 

TRAINING  &  EDUCATION  CMD 

LTCOL 

TECOM 

MODEL  SUPPORT/DESIGN 

TRAINING  &  EDUCATION  CMD 

MAJ 

TECOM 

OIC,  TECH  SUPPORT  SECTION 

TRAINING  &  EDUCATION  CMD 

MAJ 

TECOM 

SECTION  HEAD 

TRAINING  &  EDUCATION  CMD 

LTCOL 

TECOM 

EMERGING  TECH  OFFICER 

TRAINING  &  EDUCATION  CMD 

MAJ 

TECOM 

BNA  SYSTEM  SPONSOR 

TRAINING  COMMAND 

MAJ 

TECOM 

OPS/PLAN  OFF 

MCCDC  AUDIO  VISUAL 

CAPT 

MCCDC 

HEAD  PLANS  &  REQR  BR 

MOBILIZATION  BILLETS 

LTCOL 

M&RA 

CURRICULUM  COORDINATOR 

C&SC  MCU  EDU  COM 

LTCOL 

TECOM 

AC/S,  COMM  &  INFO  SYS 

MCB  CAMPEN  BASE  COMM 

COL 

l&L 

DATA  SYSTEMS  REQR  OFF 

CG  MCCDC 

MAJ 

MCCDC 
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